missing stimulus sweeps in converted abf

[tmchartrand]

For several abf files, we're seeing an issue where the converted NWB contains only response sweeps with no corresponding stimulus. In a few cases this seems to occur only for a sweep or two with the rest fine, but in most cases it seems to be all or nothing.
From debugging through the conversion, it seems that this continue is being hit every time:

x-to-nwb/src/x_to_nwb/ABF2Converter.py

Line 411 in 597e5bb

continue

It seems that somehow the abf.channelCount is too low, so the single nonzero entry in abf._dacSection.nWaveformEnable is not checked. Maybe this is an issue with pyabf?
Example abf dataset is here: https://www.dropbox.com/s/tax1mkv8tikg1zz/abf_conversion_missing_stim.zip?dl=0

[t-b]

Thanks for the report. I've looked into that.

And I think the reason that the DA data is not present in a channel is that it is created via epoch definition rather than a external stimulus file (ATF). IIRC the first ABF files I had for the conversion back in 2018 always used an external ATF file for the protocol. So this might be something which we never encountered before.

See the following screenshot from Clampex:

Can you attach a working ABF file? And also post the commandline you used to convert it?
Maybe doing some diffing reveals what is different.

I also found AllenInstitute/ipfx@c252599 which mentions that the second channel is a feedback channel of the first one and by default not outputted when converting via run_x_to_nwb_conversion.py from ipfx. Using x-to-nwb here does convert all channels by default. So we could have an escape hatch if the DA data can be not reconstructed as the feedback data might be able to replace it.

Doing some further hacking
There seems to be some confusion about the channel numbers. When I use pyabf to give me the epochs for DA 0 I get nothing but for 3 I get the epoch table.

>>> et = pyabf.waveform.EpochTable(abf, 3)
>>> print(et)
                    EPOCH         A         B         C         D         E
                     Type      Step      Step      Step      Step      Step
              First Level      0.00     10.00      0.00     10.00      0.00
              Delta Level      0.00      0.00      0.00      0.00      0.00
  First Duration (points)      2500       500      2500       500      2500
  Delta Duration (points)         0         0         0         0         0
     Digital Pattern #3-0      0000      0000      0000      0000      0000
     Digital Pattern #7-4      0000      0000      0000      0000      0000
    Train Period (points)         0         0         0         0         0
     Pulse Width (points)         0         0         0         0         0

Note to self:

• Maybe pyabf needs to visit nDACNum = [3, 3, 3, 3, 3, 3] to gather all active DA channels??

[tmchartrand]

Thanks for looking into this and sorry for the delay. I've added a second example, along with converted nwb's and nwb_to_pdf pdfs of each: https://www.dropbox.com/s/us6w1hzo3sg4rri/abf_conversion_missing_stim_complete.zip?dl=0
The converter was run via python -m ipfx.bin.run_x_to_nwb_conversion --fileType .abf --protocolDir protocols H19.29.144.11.31.01/ (I haven't switched to running from this version yet but I assume the args are equivalent).

I'll follow up with the originating lab to see if they have any insight on when they used epoch definition vs atf files, but if it's possible to extract the stimuli from the epoch definition cases without too much work that'd be great. It may be instructive to compare also to the "STEPS#0" sweeps in the working recording (H19.29.144.11.31.01), which I think also doesn't use an atf file.

Regardless, we may end up needing to use the recorded stimulus traces as you suggest in many cases anyway, due to the unrelated issue where we don't always have the correct atf files saved for each experiment.

[tmchartrand]

@t-b did you ever figure out whether these could be resolved? In particular, I think the "STEPS#0" sweeps in the working files should provide a good comparison, as mentioned above.

[t-b]

@tmchartrand Not yet sorry, currently focusing on some MIES stuff.