BucketingSampler more randomness?

[danpovey]

@pzelasko we are still seeing sawtooth patterns in losses when we use BucketingSampler, even with fewer buckets.
I think it's because the individual buckets are sorted by length. Is it possible to shuffle somehow within the buckets, or, say, always randomly pick a batch from the front or back of the bucket? Or perhaps the individual buckets could either be reversed, or not-reversed, randomly or alternately.

[csukuangfj]

https://tensorboard.dev/experiment/h0VXvnqfR8aabI3HJxHxYA/
The above tensorboard log is using bucketing sampler with the default number of buckets (30).
You can see that the attention loss jumps at the beginning of each epoch.

[pzelasko]

OK, just a sanity check — did you pass in the arguments shuffle=True to the constructor, and called sampler.set_epoch(epoch) at each epoch?

[csukuangfj]

did you pass in the arguments shuffle=True to the constructor, and called sampler.set_epoch(epoch) at each epoch?

Yes. The related code is shown below:

https://github.com/k2-fsa/icefall/blob/master/icefall/dataset/asr_datamodule.py#L172

        if self.args.bucketing_sampler:
            logging.info("Using BucketingSampler.")
            train_sampler = BucketingSampler(
                cuts_train,
                max_duration=self.args.max_duration,
                shuffle=True,
                num_buckets=self.args.num_buckets,
            )

https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/conformer_ctc/train.py#L639

    for epoch in range(params.start_epoch, params.num_epochs):
        train_dl.sampler.set_epoch(epoch)

---

The training options from the log file are:

2021-08-08 16:39:50,977 INFO [train.py:598] (0/3) 
{'exp_dir': PosixPath('conformer_ctc/exp_new'), 
'lang_dir': PosixPath('data/lang_bpe'), 
'feature_dim': 80,
'weight_decay': 1e-06, 'subsampling_factor': 4, 'start_epoch': 1, 
'num_epochs': 50, 'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 
'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 10, 
'reset_interval': 200, 'valid_interval': 3000, 'beam_size': 10, 'reduction': 'sum', 
'use_double_scores': True, 'accum_grad': 1, 'att_rate': 0.7, 'attention_dim': 512, 'nhead': 8, 
'num_decoder_layers': 6, 'is_espnet_structure': True, 'mmi_loss': False, 
'use_feat_batchnorm': True, 'lr_factor': 5.0, 'warm_step': 80000, 'world_size': 3, 
'master_port': 12354, 'tensorboard': True, 
'feature_dir': PosixPath('data/fbank'), 'max_duration': 200, 
'bucketing_sampler': True, 'num_buckets': 30, 
'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 
'on_the_fly_feats': False, 'full_libri': True}

[pzelasko]

In that case, I will double-check if the shuffling works as intended in each bucket of the BucketingSampler.

BTW I didn’t notice the drop_last=True option for BucketingSampler in what you posted — was it used?

[danpovey]

There is another factor that could cause this sawtooth pattern, which is that the buckets have the same number of samples, but you consume them in batches that depend on the duration, so the shorter buckets will be consumed first. It might be better to compute the cumulative sum of the durations, and split at percentiles of that, if that is not already what split() there does.

[pzelasko]

It doesn’t do it. I can make a PR with that option later and let’s see then.

[csukuangfj]

BTW I didn’t notice the drop_last=True option for BucketingSampler in what you posted — was it used?

I think it is unused and is left to the default value, which is False.
By the way, the drop_last is.not.exposed by lhotse, I think.

[pzelasko]

It is. See: https://github.com/lhotse-speech/lhotse/blob/master/test/dataset/test_sampling.py#L658

…

Wiadomość napisana przez Fangjun Kuang ***@***.***> w dniu 8/10/21, o godz. 08:16:  BTW I didn’t notice the drop_last=True option for BucketingSampler in what you posted — was it used? I think it is unused and is left to the default value, which is False. By the way, the drop_last is.not.exposed by lhotse, I think. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[pzelasko]

Please check out both with drop_last=True (#357) and with bucket_method='equal_duration' (#365). Let me know if any of these help.

[danpovey]

BTW, if each bucket keeps track of the amount of data it has left (however we define that , e.g. duration or samples), a relatively easy way to implement approximate proportional sampling would be to, whenever the next batch is requested, choose 2 nonempty buckets a and b, and pick from bucket a with probability (a.dur() / (a.dur() + b.dur()), else bucket b. That would ensure that all the buckets would become empty at exactly the same time. Otherwise, some of the buckets will start to deplete approximately [sqrt(num_batches_in_a_bucket) * num_buckets] batches before the end of the epoch even if they have been balanced at the start.

[pzelasko]

I'm not sure I understand why some buckets would deplete sooner. With the new method, each bucket has the same duration of speech -- since the batches are gathered to satisfy a max total duration, it implies that each bucket should yield the same number of batches. And since the buckets are chosen with uniform probabilities, they should all deplete at the same time.

Still, the method you're proposing could be useful as a source selection strategy for a "mux" sampler (that I'm planning to add soon) so that it avoids depleting its source samplers prematurely.

[pkufool]

My training crashed with batch size mismatching, there may be some bugs with bucket_method='equal_duration'.

The log showed that the returned x and mask with different batch size 52 vs 46, may be problems with subsampling and masking, I am not sure, will figure out that.

If I did not set bucket_method='equal_duration', it won't cause this crash.

https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/conformer_ctc/transformer.py#L210-L217

        x = self.encoder_embed(x)
        x = self.encoder_pos(x)
        x = x.permute(1, 0, 2)  # (N, T, C) -> (T, N, C)
        mask = encoder_padding_mask(x.size(0), supervisions)
        mask = mask.to(x.device) if mask is not None else None
        x = self.encoder(x, src_key_padding_mask=mask)  # (T, N, C)
        return x, mask 

[pzelasko]

Okay, thanks for reporting. I’m not sure what might have went wrong at the moment — I’ll try to reproduce this issue. Is this LibriSpeech 100h / 960h?

[danpovey]

Piotr: my formula was just taking into account the statistics of how random counts with equal probability are not quite the same in practice.

…

On Thu, Aug 12, 2021 at 10:28 AM Piotr Żelasko ***@***.***> wrote: Okay, thanks for reporting. I’m not sure what might have went wrong at the moment — I’ll try to reproduce this issue. Is this LibriSpeech 100h / 960h? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#364 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLOZ5ZWK2325ICKM2FF3T4MWUTANCNFSM5B3S3YAQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .