Results with pretrain_data_fraction args

[claravania]

I'm trying to train MNLI in low-data regime (~750 examples) on RoBERTa, using the pretrain_data_fraction argument. I'm using default RoBERTa hyperparameter setup, but set the num epochs higher to allow the model trained longer. However, I got quite high performance 89.02 macro_avg, just slightly lower than using full data, 90.2 macro_avg. Could there possibly be some bugs?

Attaching log file, config file, and my command.

Command:
python main.py --config_file jiant/config/nli-roberta_conf.txt -o "run_name=run1, random_seed=123456"

nli-roberta_conf.txt
log.log

(changed .conf to _conf.txt since I can't upload the format to GitHub)

[pyeres]

It looks like the task's task.train_data, which is an instance generator with a pretraining data fraction arg, is being overwritten with an instance generator with the target data fraction arg when task name appears in both the pretrain_task_names and target_task_names:

jiant/jiant/preprocess.py

Lines 421 to 436 in de3c44a

	# When using pretrain_data_fraction, we need modified iterators for use
	# only on training datasets at pretraining time.
	if task.name in pretrain_task_names:
	log.info("\tCreating trimmed pretraining-only version of " + task.name + " train.")
	task.train_data = _get_instance_generator(
	task.name, "train", preproc_dir, fraction=args.pretrain_data_fraction
	)
	pretrain_tasks.append(task)
	# When using target_train_data_fraction, we need modified iterators
	# only for training datasets at do_target_task_training time.
	if task.name in target_task_names:
	log.info("\tCreating trimmed target-only version of " + task.name + " train.")
	task.train_data = _get_instance_generator(
	task.name, "train", preproc_dir, fraction=args.target_train_data_fraction
	)
	target_tasks.append(task)

[pyeres]

Update: I'm working on a PR to address this.