Feature/unsup multichan waveform dataset

[kkarrancsu]

Added Dataset which supports multichannel audio-samples. Updated collator to drop pad-tracks.

[pzelasko]

Thanks! I think it's a good start, will take a bit more effort to handle multi-channel properly here, but we'll get there.

[pzelasko]

This should not be needed as you're manually zero-padding later.

[pzelasko]

I think there is a pitfall here, what if a MixedCut looks like:

|-------cut1-------||---padding---||----cut2----|

or any variation of the situation where padding is in between of two cuts. I don't think Lhotse would handle these situations well with your current code. Maybe you should try only removing the padding at the end (and beginning, but for that one you have to be careful about modifying the offsets on the remaining tracks). Rather than manually removing PaddingCuts, I suggest using .truncate() with carefully computed offset and duration arguments; that method will handle a lot of pitfalls and edge-cases.

[pzelasko]

Note that if you did cut.mix(musan_cut) here, it will also add an extra track; as is, the code would not work with additive noise data augmentation

[pzelasko]

I think you won't escape zero-padding of examples with less channels if you need to collate the data. However, I suggest you modify this function to return a 3-tuple: (audio, audio_lens, channel_indexes) where audio is the collated data with shape (B, C, T), audio_lens has the length of each multi-channel example of shape (B,), and channel_indexes is a list of lists of which C dim indexes have meaningful channels for examples (it could also be channel_lens tensor of shape (B,) assuming first c channels are always meaningful, if it's possible to guarantee).

[pzelasko]

But in the end your models will have to somehow deal with the non-meaningful channels anyway. As long as you're working on same-number-of-channels data, no need to overthink this.

[pzelasko]

You can ensure the tracks are sorted by some property; I imagine this is something very corpus specific and should be done by the user, not by the library.

[pzelasko]

Suggested change

	class UnsupervisedMultiChanWaveformDataset(UnsupervisedDataset):
	class MultiChannelWaveformDataset(UnsupervisedDataset):

somehow reads better to me

[pzelasko]

This line would again have the extra padding channels problem. This suggests that maybe the solution should not be (entirely) in the collate function, but inside load_audio, e.g. controlled by an extra argument?

[pzelasko]

This should be uncommented

[pzelasko]

For anybody interested in this, here's some context of our earlier discussion with @kkarrancsu

I expect you to run into issues related to padding and MUSAN data augmentation with it. Basically, padding and augmentation creates extra tracks in MixedCut, and neither MixedCut nor collate_multi_channel_audio know which tracks are the data, and which tracks are the padding / noise. So, for 4-channel audio, you might end up with 6-channel output from collate_multi_channel_audio; you might want to modify it somehow to avoid that (interpret padding as padding all channels and noise augmentation as… adding the noise to each channel? or just one of them? I don’t know)

[pzelasko]

@kkarrancsu I have a different idea -- we could add a new attribute to MixTrack that's called separate_channel: bool = True -- it would indicate if a given track is a "data" channel or an "augmentation" channel. We would add a parameter with the same name to mix (default=True) and pad (default=False) operations on all cuts.

We would need to extend FeatureMixer and AudioMixer to handle these cases properly. The idea is to again add separate_channel parameter to add_to_mix method, and modify the property unmixed_audio:

lhotse/lhotse/audio.py

Lines 984 to 990 in b41e4f8

	@property
	def unmixed_audio(self) -> np.ndarray:
	"""
	Return a numpy ndarray with the shape (num_tracks, num_samples), where each track is
	zero padded and scaled adequately to the offsets and SNR used in ``add_to_mix`` call.
	"""
	return np.vstack(self.tracks)

so that instead of simply vstacking the right channels, it vstacks only the "separate" channels, downmixes the remaining channels to mono, and adds them to each of the "separate" channels. The analogous operation is needed for FeatureMixer.

Then, collate_multi_channel_audio could remain almost unmodified w.r.t. to what's there in the codebase, and it would not need any special logic to figure out what's a real channel and what is just padding/augmentation.

Of course we'd need to add more unit tests to make sure this doesn't break anything and works as expected.

[danpovey]

It seems to me that the more "correct" way to do this would be, when adding noise to multi-channel audio, to add multiple channels of noise. I assume this would require some nontrivial simulation, possibly with multiple sources.
I would have thought that for any application that was going to process multiple-channel audio in a non-trivial way, just adding a single type of noise to all channels would not really be sufficient.

[pzelasko]

Good point.. I am not sure if implementing that on top of MixedCut makes sense though, as we would need to hold all the information relating to the "nontrivial simulation" in the manifests. It might make more sense to write a dedicated module/transform that works directly on audio data (or use an existing tool).

One such tool is e.g. https://github.com/asteroid-team/torch-audiomentations, but I just noticed that they are doing exactly the same simplified mono downmix I was thinking about:

https://github.com/asteroid-team/torch-audiomentations/blob/261015b3fdd99b475507aab01456093e13719519/torch_audiomentations/augmentations/background_noise.py#L13-L21

Another option is using https://github.com/LCAV/pyroomacoustics as a transform inside your PyTorch Dataset class I think.

In any case, we would still need to be able to handle the padding. I think the solution I suggested with separate_channel is complimentary to using more advanced simulations later in the data pipeline (e.g., manifests only contain the padding information, MixedCut loads the audio and doesn't add the extra channel for padding, and noise+simulation is added via a transform inside Dataset).

[danpovey]

OK, sure, it was just a thought.