Using jiant to make BioBERT and SciBERT to run superGLUE tasks

[shi-kejian]

Hi,
As always, thank you for this amazing contribution.

I am taking prof.bowman's class and attempting to run BioBERT https://github.com/dmis-lab/biobert-pytorch and SciBERT https://github.com/allenai/scibert on jiant. One of our objectives for the course project is to run BioBERT and SciBERT on common NLU tasks.

As far as I understand, it should be possible to add any Transformers encoder model to jiant, but both of those models will probably require a bit of code?

I am sketching out what I'd have to do in jiant vs. plain transformers. Will using jiant create more overheads (by trying to support those models) than just following the standard fine-tuning processes in transformers?

Any suggestions and pointers will be helpful. Thank you!

[jeswan]

I think either fine-tuning with transformers or using jiant is suitable for your task. If you decide to use jiant, I will guide you through the process. BioBERT and SciBERT are available as pertained models here: https://huggingface.co/models. Both of the models you mentioned appear to have a BERT architecture with different pretraining objectives. I am currently working on a feature branch (https://github.com/nyu-mll/jiant/tree/js/feature/easy_add_model - should be merged to main in the next week) that is suitable for your goal. Working off the feature branch would be a good place to start. You can simply use the model name from the Transformers Model Hub as an argument(hf_pretrained_model_name_or_path="allenai/scibert_scivocab_cased") in the simple API to fine-tune the models.

What tasks are you planning on running? The following notebook is an example of how to fine tune tasks currently supported by jiant: https://colab.research.google.com/github/nyu-mll/jiant/blob/master/examples/notebooks/simple_api_fine_tuning.ipynb

List of supported tasks: https://github.com/nyu-mll/jiant/blob/master/guides/tasks/supported_tasks.md

[shi-kejian]

Thank you very much Jesse for the prompt reply!

Yeah, I think using jiant will definitely save us lots of code and time.

For the mllu course project, we aim to measure the degree to which using a different pre-training corpus would change that affect both in terms of the amount of bias learned and used in downstream applications but also what is the tradeoff with actual pure NLU performance. I am planning to run BERT, RoBERTa, BioBERT, and SciBERT on ALL SuperGLUE tasks, because I don't think there is a way to clearly say apriori which tasks would be best at discriminating the models.

Back to my question. I guess the main issue here is that I'd better finish the tasks by the end of this week because there will be a partial draft due next Thursday, so I am afraid I will not be able to use your amazing new feature :(. However, by looking at your notebook examples, I think I can also use the current main branch of jiant right? by loading different models in line say MODEL_TYPE = "roberta-base". Or did I miss anything?

Again, thank you very much. And it would be much appreciated if I could ask more follow-up questions during the process (I am planning to install jiant on NYU greene and run the tasks). @jeswan

edit: sorry I accidentally removed myself as an assignee for this thread. If you need it, please reassign me (I guess that's how you keep track of the issues?) Thank you so much. @jeswan

[jeswan]

jiant would be well-suited for what you described. You could use the main branch of jiant but may need to make some modifications for jiant to recognize SciBERT as a BERT model. As a starting point, I would use this notebook: https://colab.research.google.com/github/nyu-mll/jiant/blob/master/examples/notebooks/jiant_Basic_Example.ipynb. You will need to change the following:

export_model.export_model(
    hf_pretrained_model_name_or_path="allenai/scibert_scivocab_cased",
    output_base_path="./models/scibert",
)

In the subsequent steps, you will need to modify jiant to treat allenai/scibert_scivocab_cased the same as bert (this is not required in the feature branch mentioned above). Feel free to ask any additional questions during the process.

I previously assigned the issue to you to notify you that I required your feedback. For now, I'll close this issue but please reopen the issue and comment if you have more questions.

[shi-kejian]

Thank you very much for the resource.

I would like to quickly follow up tho. When you mentioned "to modify jiant", did you mean we need to somehow modify the source code of jiant, after we install jiant on Greene HPC? Or do we just need to modify, say, the example notebook to make sure all configs are correct for each model and task?
Right now I am setting up jiant (master branch) on HPC.
Thank you for your time.

edit: Or maybe I just install the feature branch?

[jeswan]

Modify the source code. However, if you use the feature branch above (should be merged into master within a week), you should not need to modify the source code.

[shi-kejian]

Thank you.
I think I can first run BERT and RoBERTa (already supported) using the simple fine-tuning API and get the results.
Since that's just a partial draft due next week plus we had other finished bias detecting experiments, so I think I can leave running BioBERT and SciBERT on SuperGLUE later next week, using your amazing new feature.

Thanks again.

[shi-kejian]

Hello, I actually run into a problem. I installed jiant v2 on HPC Greene. And I am running the simple API using python. I am running roberta-large on superglue tasks. However, it says that superglue_axg and superglue_axb are not supported? Here is the screenshot:

Thank you very much.

[jeswan]

Please use superglue_broadcoverage_diagnostics and superglue_winogender_diagnostics. The following discussion may also be relevant: #1255

[shi-kejian]

Thanks for the prompt reply!

[shi-kejian]

Dear Jesse,
Just checking in -- will the easy_add_model branch be merged to master recently as expected?
We were simply using the Simple API.
.....
..... (previous information deprecated)
Thank you for the amazing tool!! @jeswan

Update: Sorry I think I was confused! So previously you actually meant the feature branch IS WORKING right? I just need to download that branch not the master... So I don't need to wait till it merged to main.

Update: I was working on the easy_add_model feature branch just using Simple API. However, it seems like the data path was not compatible with the current master branch.

Here are the details:
I have a directory called "biobert_base" inside /jiant.
And here is my python code using Simple API to run Biobert on BoolQ.

Downloading the data and model seems good:

However, right before it starts to tokenize: I have an error:

So it seems like somewhere the path gets concatenated twice? I am trying to locate which files are deprecated. If this sounds/looks familiar to you, any pointer would be appreciated. Thank you so much!