Video Time Encoding and Decoding Machines¶

This package provides code to encode and decode videos with time encoding machines consisting of Gabor or center-surround receptive fields followed by Integrate-and-fire neurons [1] , [2] . It supports both the pseudoinverse algorithm, described in [1] , [2] and recurrent neural networks method, described in [3] for decoding.

Additional time encoding and decoding algorithms implemented in Python are available in the Time Encoding and Decoding (TED) Toolkit.

Contents¶

Installation Instructions¶

Quick Installation¶

If you have pip installed, you should be able to install the latest stable release of vtem by running the following:

pip install vtem

All dependencies should be automatically downloaded and installed if they are not already on your system.

Obtaining the Latest Software¶

The latest stable and development versions of vtem can be downloaded from GitHub

Online documentation for vtem is available at http://bionet.github.io/vtem/index.html/

Installation Dependencies¶

vtem requires that the following software packages be installed:

Python 2.7 or later.
setuptools 0.6c10 or later.
NumPy 1.6.0 or later.
CUDA SciKit 0.042 or greater
PyTables 2.4.0 or greater
PyCUDA 2011.1 or later
NVIDIA CUDA Toolkit 4.0 or later.

To support reading and writing video files directly, the package requires

OpenCV with python and FFmpeg support

To build the documentation, the following packages are also required:

Sphinx 1.1.0 or later.

Running the demos¶

To run the demos in the package as it is, please download the demo files and unarchive it in the demos folder.

To run the demos on other videos, the filename can be provided with the -i switch. All options supported can be viewed by:

python <demo_filename> --help

If you have OpenCV installed, all video files encoded with codecs that have been installed with OpenCV should be supported.

To run without OpenCV support, please load the video into MATLAB and save it using the scripts provided in the matlab_h5 folder of the package.

Platform Support¶

The software has been developed and tested on Linux; it should also work on other Unix-like platforms supported by the above packages. Parts of the package may work on Windows as well, but remain untested.

Building and Installation¶

To build and install the toolbox, download and unpack the source release and run:

python setup.py install

from within the main directory in the release. To rebuild the documentation, run:

python setup.py build_sphinx

Getting Started¶

Sample codes for the package are located in the /demos subdirectory. For a detailed description of the package modules, please refer the reference section of the package documentation.

Reference¶

vtem.vtem¶

This module provides the functions to encode a video using time encoding machines and store the resultant spikes and also the parameters of the VTEM in h5 files.

`VTEM_IAF`	Encode a video with IAF neurons and the specified Receptive field
`VTEM_Gabor_IAF`	Encode a video with IAF neurons and gabor receptive field
`VTEM_CS_IAF`	Encode a video with IAF neurons and gabor receptive field

vtem.vtdm¶

This module provides the functions to recover a video enocded using time encoding machines.

decode_video Reconstruct video using VTDM with dirichlet kernel, assuming IAF neurons

vtem.ensemble_encode¶

This module provides the functions to encode the output of a receptive field using an ensemble of neurons. Only Integrate and fire neurons are supported in the current release

IAF_encode¶

class vtem.ensemble_encode.IAF_encode(num_neurons, dt, dtype=<type 'numpy.float64'>)¶

Methods

`encode`(neural_inputs[, startbias, avg_rate])	Encode with IAFs
`load_parameters`([h_kappa, h_bias, h_delta, ...])	Load encoding parameters to GPU
`reset_timer`()	Reset the time_count to zeros
`set_initial_value`()	Set integration from zeros
`set_parameters`()	Set the parameters using default values
`set_time_count`()	Set time_count zeros

__init__(num_neurons, dt, dtype=<type 'numpy.float64'>)¶

Population encoding with IAF neurons

Parameters:

num_neurons : integer

number of neurons to encode

dt : float

time interval between two consecutive samples in the input for each neuron

dtype : np.float64 or np.float32, optional

dtype of spike interval to be stored (np.float32 not yet tested) If not specified, will be set to np.float64

encode(neural_inputs, startbias=0, avg_rate=0.1)¶

Encode with IAFs

Parameters:

neural_inputs : PitchArray

PitchArray of shape (num_samples, num_neurons) containing inputs to all neurons

startbias : integer, optional

the neuron index corresponding to first column of neural_inputs

avg_rate: float, optional

average spiking rate assumed for neurons, will allocate memory num_samples/avg_rate for each neuron for storing spikes

If not specified, will be set to 0.1

Returns:

spikes: ndarray of self.dtype

stores the spikes for one neuron after another

spike_count: ndarray of int32 of size num_neurons

indicates the number of spikes generated by each neuron

Notes

spikes for neuron j can be accessed by

cum_count = np.concatenate((np.zeros(1,np.int32),np.cumsum(spike_count)))
tk = spikes[cum_count[j]:cum_count[j+1]]

load_parameters(h_kappa=None, h_bias=None, h_delta=None, h_sigma=None, h_time_count=None, h_v0=None)¶

Load encoding parameters to GPU

h_kappa, h_bias, h_delta can be set up using default values (using None), or specified together (using ndarrays of dtype)

h_time_count can be set up using default values (using None), or specified values(using ndarrays of dtype)

h_v0 can be set up using default values (using None), or specified values(using ndarrays of dtype)

reset_timer()¶: Reset the time_count to zeros

set_initial_value()¶: Set integration from zeros

set_parameters()¶

Set the parameters using default values

Will set kappa’s to be 1.0, deltas to be 0.03, bias to be 0.8

set_time_count()¶: Set time_count zeros

IAF_encode_rt¶

class vtem.ensemble_encode.IAF_encode_rt(num_neurons, dt, dtype=<type 'numpy.float64'>)¶

Methods

`encode`(neural_inputs[, startbias, avg_rate])	Encode with IAFs with random thresholds
`load_parameters`([h_kappa, h_bias, h_delta, ...])	Load encoding parameters to GPU
`reset_timer`()	reseting the time_count to zeros
`set_initial_value`()	set integration from zeros
`set_parameters`()	Set the parameters using default values
`set_time_count`()	Set time_count to zeros

__init__(num_neurons, dt, dtype=<type 'numpy.float64'>)¶

Population encoding with IAF neurons with random thresholds

Parameters:

num_neurons : integer

number of neurons to encode

dt : float

time interval between two consecutive samples in the input for each neuron

dtype : type, optional

dtype of spike interval to be stored ( np.float32 not tested)

encode(neural_inputs, startbias=0, avg_rate=0.1)¶

Encode with IAFs with random thresholds

Parameters:

neural_inputs : PitchArray

PitchArray of shape (num_samples, num_neurons) containing inputs to all neurons

startbias : integer

the neuron index corresponding to first column of neural_inputs

avg_rate : float

average spiking rate assumed for neurons, will allocate memory num_samples/avg_rate for each neuron for storing spikes

Returns:

spikes : ndarray of self.dtype

stores the spikes for one neuron after another

spike_count : ndarray of int32 of size num_neurons

indicates the number of spikes generated by each neuron

Notes

spikes for neuron j can be accessed by

cum_count = np.concatenate((np.zeros(1,np.int32),np.cumsum(spike_count)))
tk = spikes[cum_count[j]:cum_count[j+1]]

load_parameters(h_kappa=None, h_bias=None, h_delta=None, h_sigma=None, h_delta_value=None, h_time_count=None, h_v0=None)¶

Load encoding parameters to GPU

h_kappa, h_bias, h_delta can be set up using default values (using None), or specified together (using ndarrays of dtype)

h_time_count can be set up using default values (using None), or specified values(using ndarrays of dtype)

h_v0 can be set up using default values (using None), or specified values(using ndarrays of dtype)

reset_timer()¶: reseting the time_count to zeros

set_initial_value()¶: set integration from zeros

set_parameters()¶

Set the parameters using default values

Will set kappa’s to be 1.0, deltas to be 0.03, bias to be 0.8

set_time_count()¶: Set time_count to zeros

vtem.vrf¶

This module provides the functions to compute the output at the receptive field when presented with a video.

vrf_gabor¶

class vtem.vrf.vrf_gabor(shape, dtype=<type 'numpy.float64'>, dx=0.0625, dy=0.0625, scale=1, domain=None)¶

Gabor RFs

Methods

`compute_Ds`(Mx, My)	Returns the dirichlet coefficients of all gabor filters with order Mx, My
`compute_Dsw`(d_Ds, Mx, My, h_norm)	Compute the weighting matrix of the “correlation” between each two RFs
`compute_dirich_space`(d_Ds, Mx, My, Px, Py, ...)	Compute the spatial reconstruction functions
`compute_dirich_space_fft`(d_Ds, Mx, My, Px, ...)
`filter`(V)	Filter a video V
`generate_visual_receptive_fields`([...])	Generate a batch of gabor filters from parameters set in self
`get_gabor_parameters`([dilations, ...])	Create a set of parameters for a family of gabor filters
`load_parameters`([num_neurons, h_alpha, h_l, ...])	Load gabor parameters to GPU

__init__(shape, dtype=<type 'numpy.float64'>, dx=0.0625, dy=0.0625, scale=1, domain=None)¶

Parameters:

shape : list

2-tuple indicating the shape (number of pixels in (height,width) / (y,x) ) of RF

dtype : type, optional

numpy dtype to use for RF calculations. If not specified , will default to np.float64

dx : float

spatial resolution in x direction, distance between two pixels

dy: float

spatial resolution in y direction, distance between two pixels

scale : float

scale up the shape by scale and scale down dx,dy by scale effectively increase the resoltuion of the RFs

domain : list

list of 2, [a,b], specifying the domain to cover RFs in default parameter settings a in x direction, b in y direction

Notes

The coordinate system is given by the following:

row (width / X) major

-----------------------------------------> width
|   Y
|   ^-------------------->
|   |-------------------->
|   |-------------------->
|   |--------------------> X
| 
|
v
height

compute_Ds(Mx, My)¶: Returns the dirichlet coefficients of all gabor filters with order Mx, My in the format of PitchArray with shape (num_neurons, 2*Mx+1, 2*My+1)

compute_Dsw(d_Ds, Mx, My, h_norm)¶

Compute the weighting matrix of the “correlation” between each two RFs

Parameters:

d_Ds : PitchArray

containing dirichlet coefficient most possibly created by compute_Ds

Mx : integer

order in the x dimension

My : integer

order in the y dimension

Returns:

PitchArray with shape (num_neurons, num_neurons)

compute_dirich_space(d_Ds, Mx, My, Px, Py, Sx, Sy, Wx, Wy, x_start=None, y_start=None)¶

Compute the spatial reconstruction functions

Parameters:

d_Ds : PitchArray

containing dirichlet coefficient most possibly created by compute_Ds

Mx : integer

Order in the x dimension

My : integer

Order in the y dimension

Px : integer

number of pixels in reconstruction functions in the x dimension

Py : integer

number of pixels in reconstruction functions in the y dimension

Sx : float

spatial domain in degree of reconstruction functions in x direction

Sy : float

spatial domain in degree of reconstruction functions in y direction

Wx : float

spatial bandwidth in x direction

Wy : float

spatial bandwidth in y direction

x_start : float

indicating the starting degree in x direction

y_start : float

indicating the starting degree in y direction

output: PitchArray with shape (num_neurons, Px, Py)

filter(V)¶

Filter a video V Must set up parameters of gabor first

Parameters:

V : 3D ndarray, with shape (num_frames, Px, Py)

Returns:

The filtered output by the gabor filters specified in self

output is a PitchArray with shape (num_neurons, num_frames)

jth row of which is the output of jth gabor filter

generate_visual_receptive_fields(startbias=0, N_filters=None, x_start=None, y_start=None)¶

Generate a batch of gabor filters from parameters set in self

Parameters:

start_bias : integer, optional

start the the (start_bias)th filter

N_filters : integer, optional

number of filters to generate

x_start : float, optional

indicating the starting degree in x direction

y_start : float, optional

indicating the starting degree in y direction

get_gabor_parameters(dilations=[1, 0, -1, -2, -3], num_rotations=8, alpha0=2.0, b0=1.5, b0scalebase=1.4142135623730951, KAPPA=2.5)¶

Create a set of parameters for a family of gabor filters

Parameters:

dilation : list

a list containing the dilation parameter m

num_rotations : integer

number of rotations of each (dilation, translation_x, translation_y) pair

alpha0 : np.float64

base factor of dilation

b0 : np.float64

base factor of translation

b0scalebase : np.float64

effective translation in each dilation will be b0 * (alpha0**dilations[i]) * b0scalebase**(-dilations[i])

KAPPA : float

spatial bandwidth

load_parameters(num_neurons=None, h_alpha=None, h_l=None, h_x0=None, h_y0=None, h_ab=None, KAPPA=2.5, set=0)¶

Load gabor parameters to GPU

num_neurons, h_alpha, h_l, h_x0, h_y0, h_ab must be specified together or not specified at all

Parameters:

num_neurons : integer

total number of neurons

h_alpha : ndarray of float64

containing dilation parameters alpha = alpha0**(m)

h_l : ndarray of float64

containing rotation parameters l (angles)

h_x0 : ndarray of float64

containing translation parameters x0

h_y0 : ndarray of float64

containing translation parameters y0

h_ab : ndarray of int32

containing 0 or 1, 0 for real part, 1 for imaginary part jth gabor filter will be generated according to h_alpha[j], h_l[j], h_n[j], h_k[j] and h_ab[j]

KAPPA : float

spatial bandwidth

set : integer

0 if self parameters has not been set, 1 if they have been set by get_gabor_parameters or other manner)

vrf_cs¶

class vtem.vrf.vrf_cs(shape, dtype=<type 'numpy.float64'>, dx=0.0625, dy=0.0625, scale=1, domain=None)¶

Centre Surround RFs

Methods

compute_Ds(Mx, My)

Parameters:

compute_Dsw(d_Ds, Mx, My, h_norm) Compute the weighting matrix of the “correlation” between each two RFs

compute_dirich_space(d_Ds, Mx, My, Px, Py, ...) Compute the spatial reconstruction functions

compute_dirich_space_fft(d_Ds, Mx, My, Px, ...)

filter(V) Filter a video V

generate_visual_receptive_fields([...]) Generate a batch of centre surround filters from parameters set in self

get_cs_parameters([dilations, alpha0, b0, ...]) Create a set of parameters for a family of Centre Surround filters

load_parameters([num_neurons, h_alpha, ...]) Load Centre Surround parameters to GPU

__init__(shape, dtype=<type 'numpy.float64'>, dx=0.0625, dy=0.0625, scale=1, domain=None)¶

Parameters:

shape : list

2-tuple indicating the shape (number of pixels in (height,width) / (y,x) ) of RF

dtype : type, optional

numpy dtype to use for RF calculations. If not specified , will default to np.float64

dx : float

spatial resolution in x direction, distance between two pixels

dy: float

spatial resolution in y direction, distance between two pixels

scale : float

scale up the shape by scale and scale down dx,dy by scale effectively increase the resoltuion of the RFs

domain : list

list of 2, [a,b], specifying the domain to cover RFs in default parameter settings a in x direction, b in y direction

Notes

The coordinate system is given by the following:

row (width / X) major

-----------------------------------------> width
|   Y
|   ^-------------------->
|   |-------------------->
|   |-------------------->
|   |--------------------> X
| 
|
v
height

compute_Ds(Mx, My)¶

Parameters:

Mx : integer

Order in the x dimension

My : integer

Order in the y dimension

Returns:

The dirichlet coefficients of all gabor filters with order Mx, My

in the format of PitchArray with shape (num_neurons, 2*Mx+1, 2*My+1)

compute_Dsw(d_Ds, Mx, My, h_norm)¶

Compute the weighting matrix of the “correlation” between each two RFs

Parameters:

d_Ds : PitchArray

containing dirichlet coefficient most possibly created by compute_Ds

Mx : integer

order in the x direction

My : integer

order in the y direction

Returns:

PitchArray with shape (num_neurons, num_neurons)

compute_dirich_space(d_Ds, Mx, My, Px, Py, Sx, Sy, Wx, Wy, x_start=None, y_start=None)¶

Compute the spatial reconstruction functions

Parameters:

d_Ds : PitchArray

containing dirichlet coefficient most possibly created by compute_Ds

Mx : integer

Order in the x direction

My : integer

Order in the y direction

Px : integer

number of pixels in reconstruction functions in x direction

Py : integer

number of pixels in reconstruction functions in y direction

Sx : float

spatial domain in degree of reconstruction functions in x direction

Sy : float

spatial domain in degree of reconstruction functions in y direction

Wx : float

spatial bandwidth in x direction

Wy : float

spatial bandwidth in y direction

x_start : float, optional

indicating the starting degree in x direction

y_start : float, optional

indicating the starting degree in y direction

output: PitchArray with shape (num_neurons, Px, Py)

filter(V)¶

Filter a video V Must set up parameters of CS RF first

Parameters:

V : 3D ndarray, with shape (num_frames, Px, Py)

Returns:

the filtered output by the gabor filters specified in self

output is a PitchArray with shape (num_neurons, num_frames),

jth row of which is the output of jth gabor filter

generate_visual_receptive_fields(startbias=0, N_filters=None, x_start=None, y_start=None)¶

Generate a batch of centre surround filters from parameters set in self

Parameters:

start_bias : integer, optional

start the the (start_bias)th filter

N_filters : integer, optional

generate N_filters filters

x_start : float

indicating the starting degree in x direction

y_start : float

indicating the starting degree in y direction

get_cs_parameters(dilations=[1, 0, -1, -2, -3], alpha0=2.0, b0=1.0, b0scalebase=1.0, sigma_center=0.5, sigma_surround=0.8)¶

Create a set of parameters for a family of Centre Surround filters

Parameters:

dilation : list, optional

a list containing the dilation parameter m

num_rotations : list, optional

number of rotations of each (dilation, translation_x, translation_y) pair

alpha0 : np.float64, optional

base factor of dilation

b0 : np.float64, optional

base factor of translation

b0scalebase : np.float64, optional

effective translation in each dilation will be b0 * (alpha0**dilations[i]) * b0scalebase**(-dilations[i])

sigma_center : float, optional

standard deviation of the center

sigma_surround : float, optional

standard deviation of the surround

load_parameters(num_neurons=None, h_alpha=None, h_x0=None, h_y0=None, set=0, sigma_center=0.5, sigma_surround=0.8)¶

Load Centre Surround parameters to GPU

num_neurons, h_alpha, h_l, h_x0, h_y0, h_ab must be specified together or unspecified together

Parameters:

num_neurons : integer

total number of neurons

h_alpha : ndarray of float64

containing dilation parameters alpha = alpha0**(m)

h_l : ndarray of float64

containing rotation parameters l (angles)

h_x0 : ndarray of float64

containing translation parameters x0

h_y0 : ndarray of float64

containing translation parameters y0

h_ab : ndarray of int32

containing 0 or 1, 0 for real part, 1 for imaginary part jth gabor filter will be generated according to h_alpha[j], h_l[j], h_n[j], h_k[j] and h_ab[j]

set : integer

0 if self parameters has not been set, 1 if they have been set by get_gabor_parameters or other manner)

sigma_center : float, optional

standard deviation of the center

sigma_surround : float, optional

standard deviation of the surround

Authors & Acknowledgments¶

This software was written and packaged by Nikul H. Ukani and Yiyin Zhou, currently at the Bionet Group [1] at Columbia University, under the supervision of Prof. Aurel A. Lazar.

[1]	http://bionet.ee.columbia.edu/

License¶

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of Nikul H. Ukani, Yiyin Zhou, nor the names of any contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Change Log¶

Release 0.1.1 - (June 7, 2013)¶

First public release.

Index¶

[1]	(1, 2) Video Time Encoding Machines, Aurel A. Lazar and Eftychios A. Pnevmatikakis, IEEE Transactions on Neural Networks, Volume 22 , Number 3 , pp. 461-473 , March 2011

[2]	(1, 2) Encoding Natural Scenes with Neural Circuits with Random Thresholds, Aurel A. Lazar, Eftychios A. Pnevmatikakis and Yiyin Zhou, Vision Research, Volume 50, Number 22, pp. 2200-2212, October 2010, Special Issue on Mathematical Models of Visual Coding

[3]	Massively Parallel Neural Encoding and Decoding of Visual Stimuli, Aurel A. Lazar and Yiyin Zhou, Neural Networks, Volume 32, pp. 303-312, August 2012, Special Issue: IJCNN 2011