In the recent years, transformer-based language learning models like BERT have been one of the most popular architectures used in the research related to natural lan- guage processing. Productionizing these models under constrained resources such as edge computing require smaller versions of BERT like DistilBERT. However, it is a concern that these models have inadequate robustness against adversarial examples and attacks. This paper evaluates the performance of various models built on the ideas of adversarial training and GAN BERT finetuned on SST-2 dataset. Further the experiments in this paper seek to find evidence on whether knowledege distillation preserves robustness in the student models.
Vijay Kalmath Department of Data Science [email protected]
Amrutha Varshini Sundar Department of Data Science [email protected]
Sean Chen Department of Computer Science [email protected]
Google Drive Link to Models : https://drive.google.com/drive/folders/1q1TGTOl6BftZzn1ZidzvN8GKVRbTdrjM?usp=sharing