Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1882–1894 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1882 Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks Carlos Aspillaga*†, Andr´es Carvallo†‡, Vladimir Araujo*†‡ †Pontificia Universidad Cat´olica de Chile, Santiago, Chile ‡IMFD, Santiago, Chile {cjaspill, afcarvallo, vgaraujo}@uc.cl Abstract There has been significant progress in recent years in the field of Natural Language Processing thanks to the introduction of the Trans- former architecture. Current state-of-the-art models, via a large number of parameters and pre-training on massive text corpus, have shown impressive results on several downstream tasks. Many researchers have studied previous (non-Transformer) models to understand their actual behavior under different scenarios, showing that these models are taking advantage of clues or failures of datasets and that slight perturbations on the input data can severely reduce their performance. In contrast, recent models have not been systematically tested with adversarial-examples in order to show their robustness under severe stress conditions. For that reason, this work evaluates three Transformer-based models (RoBERTa, XLNet, and BERT) in Natural Language Inference (NLI) and Question Answering (QA) tasks to know if they are more robust or if they have the same flaws as their predecessors. As a result, our experiments reveal that RoBERTa, XLNet and BERT are more robust than recurrent neural network models to stress tests for both NLI and QA tasks. Nevertheless, they are still very fragile and demonstrate various unexpected behaviors, thus revealing that there is still room for future improvement in this field. Keywords: adversarial evaluation, stress tests, natural language inference, natural language understanding, question answering 1. Introduction Deep learning has allowed for solving several problems re- lated to natural language processing (NLP), even outper- forming human performance in some tasks, such as multi- label classification (Tsoumakas and Katakis, 2007), doc- ument screening (Carvallo and Parra, 2019), named en- tity recognition (Nadeau and Sekine, 2007), among others. However, previous research has shown that neural networks are powerful enough to memorize the training data, which limits their ability to generalize or to really understand the tasks they are dealing with (Zhang et al., 2017). One way to test NLP models is by using adversarial tests, which implies an intentional perturbation of the input sen- tence to confuse a model into making wrong predictions. This methodology has shown that models are still weak (Belinkov and Bisk, 2018; Iyyer et al., 2018; Ribeiro et al., 2018; Ebrahimi et al., 2018). Other researchers have also shown that language models can “falsely” solve the task. In other words, they might be taking advantage of dataset failures or artifacts on the input sentences in order to guess the answer (Gururangan et al., 2018; Agrawal et al., 2016; Levy et al., 2015). These evaluations, also known as “stress tests”, have been performed on classic models based on recurrent networks (RNN). However, Transformer-based models such as RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019) and BERT (Devlin et al., …

Content truncated to 500 words