bug in the dataloader demo

[urancon]

Hi, new experanto user here, and first of all thanks a lot for maintaining this repo, it looks really useful and I really like the project !
I've been trying to run the basic example notebooks, but stumbled upon what I find is a bug on examples/demo.ipynb. I think it can even be seen on the preview on github.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[7], line 8
      5 for k, v in batch.items():
      6     # print(f"modality: {k}, shape: {v.shape}")
      7     print("Modality: ", k)
----> 8     print("Shape: ", v.shape)
     10 # video shape: batch, times, channels, height, width
     11 # neuronal responses: batch, times, neurons

AttributeError: 'dict' object has no attribute 'shape'

Basically the "timestamps" item out of the dataloader is a dict with one key for each modality (here "screen" and "responses") and a numpy array for each of those, which by the way look almost practically the same. This is why we can't take the shape out of this variable.

Importantly, this bug seems to also appear during the instantiation of demo models with make_video_model(...).
Eg, the following lines from the allen-exporter/examples/example_training.ipynb

...

from sensorium.models.make_model import make_video_model

factorised_3d_model = make_video_model(
    data_loaders,
    seed,
    core_dict=factorised_3D_core_dict,
    core_type='3D_factorised',
    readout_dict=readout_dict.copy(),
    readout_type='gaussian',               
    use_gru=False,
    gru_dict=None,
    use_shifter=False,
    shifter_dict=shifter_dict,
    shifter_type='MLP',
    deeplake_ds=False,
)

...

will give the following traceback:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[13], line 1
----> 1 factorised_3d_model = make_video_model(
      2     data_loaders,
      3     seed,
      4     core_dict=factorised_3D_core_dict,
      5     core_type='3D_factorised',
      6     readout_dict=readout_dict.copy(),
      7     readout_type='gaussian',               
      8     use_gru=False,
      9     gru_dict=None,
     10     use_shifter=False,
     11     shifter_dict=shifter_dict,
     12     shifter_type='MLP',
     13     deeplake_ds=False,
     14 )

File [~/PycharmProjects/sensorium_2023/sensorium/models/make_model.py:68](http://localhost:8888/home/ulysse/PycharmProjects/sensorium_2023/sensorium/models/make_model.py#line=67), in make_video_model(dataloaders, seed, core_dict, core_type, readout_dict, readout_type, use_gru, gru_dict, use_shifter, shifter_dict, shifter_type, elu_offset, nonlinearity_type, nonlinearity_config, deeplake_ds)
     63 batch = next(iter(list(dataloaders.values())[0]))
     64 in_name, out_name = (
     65     list(batch.keys())[:2] if isinstance(batch, dict) else batch._fields[:2]
     66 )
---> 68 session_shape_dict = get_dims_for_loader_dict(dataloaders, deeplake_ds)
     69 n_neurons_dict = {k: v[out_name][1] for k, v in session_shape_dict.items()}
     70 input_channels = [v[in_name][1] for v in session_shape_dict.values()]

File [~/PycharmProjects/sensorium_2023/sensorium/models/utility.py:52](http://localhost:8888/home/ulysse/PycharmProjects/sensorium_2023/sensorium/models/utility.py#line=51), in get_dims_for_loader_dict(dataloaders, deeplake_ds)
     40 def get_dims_for_loader_dict(dataloaders, deeplake_ds):
     41     """
     42     ## adopted from nnfabrik - https://github.com/sinzlab/nnfabrik/blob/master/nnfabrik/utility/nn_helpers.py
     43     Given a dictionary of DataLoaders, returns a dictionary with same keys as the
   (...)
     50         dict: A dict containing the result of calling `get_io_dims` for each entry of the input dict
     51     """
---> 52     return {k: get_io_dims(v, deeplake_ds) for k, v in dataloaders.items()}

File [~/PycharmProjects/sensorium_2023/sensorium/models/utility.py:52](http://localhost:8888/home/ulysse/PycharmProjects/sensorium_2023/sensorium/models/utility.py#line=51), in <dictcomp>(.0)
     40 def get_dims_for_loader_dict(dataloaders, deeplake_ds):
     41     """
     42     ## adopted from nnfabrik - https://github.com/sinzlab/nnfabrik/blob/master/nnfabrik/utility/nn_helpers.py
     43     Given a dictionary of DataLoaders, returns a dictionary with same keys as the
   (...)
     50         dict: A dict containing the result of calling `get_io_dims` for each entry of the input dict
     51     """
---> 52     return {k: get_io_dims(v, deeplake_ds) for k, v in dataloaders.items()}

File [~/PycharmProjects/sensorium_2023/sensorium/models/utility.py:35](http://localhost:8888/home/ulysse/PycharmProjects/sensorium_2023/sensorium/models/utility.py#line=34), in get_io_dims(data_loader, deeplake_ds)
     33 else:
     34     if hasattr(items, "items"):  # if dict like
---> 35         return {k: v.shape for k, v in items.items()}
     36     else:
     37         return (v.shape for v in items)

File [~/PycharmProjects/sensorium_2023/sensorium/models/utility.py:35](http://localhost:8888/home/ulysse/PycharmProjects/sensorium_2023/sensorium/models/utility.py#line=34), in <dictcomp>(.0)
     33 else:
     34     if hasattr(items, "items"):  # if dict like
---> 35         return {k: v.shape for k, v in items.items()}
     36     else:
     37         return (v.shape for v in items)

AttributeError: 'dict' object has no attribute 'shape'

What is the expected format for the timestamps out of the dataloader ? A single array to which other modalities are aligned ? Could we try and fix this bug ?
Or is there another stable and working version of experanto/sensorium I could go back to (at least temporarily) ?

Thanks for your help !

[pollytur]

Hi @urancon,
Thanks for the catch, we will fix it in the example - would you mind submitting a PR changing the notebook?

As for your question - timestamps are currently a dict in the ChunkDataset (see __getitem__ here)
If the sampling rates of the interpolated modalities are close to be same - the timestamps would be close to be same. However, if sampling rates are different - the timestamps would be different as well.
The Dataset entity is a project related code and could be changed depending on the needs on the user, so timestamps might be removed completely if needed.

For the sake of using the sensorium base model - that model architecture requires that all of the devices are set to have equal (same) sampling rates. Moreover, sensorium models do not require timestamps as batch output at all - hence, you can just exclude timestamps from the out_keys in the config and this should solve the problem

Lmk if this helps / answers your question

PS The notebook would also benefit from the explanation like this one above

[urancon]

Thanks for your reply, it clarifies things a lot.
I proposed some changes to the notebook to fix the bug and include a tiny bit more context (i.e., what you said in your message).
Thanks again for your help.