On the Robustness of Language Encoders against Grammatical Errors

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3386–3403 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3386 On the Robustness of Language Encoders against Grammatical Errors Fan Yin1, Quanyu Long2, Tao Meng3, and Kai-Wei Chang3 1Peking University 2Shanghai Jiao Tong University 3University of California, Los Angeles 1600012975@pku.edu.cn; oscar.long@sjtu.edu.cn; tmeng@cs.ucla.edu; kwchang@cs.ucla.edu Abstract We conduct a thorough study to diagnose the behaviors of pre-trained language en- coders (ELMo, BERT, and RoBERTa) when confronted with natural grammatical errors. Specifically, we collect real grammatical er- rors from non-native speakers and conduct ad- versarial attacks to simulate these errors on clean text data. We use this approach to facil- itate debugging models on downstream appli- cations. Results confirm that the performance of all tested models is affected but the degree of impact varies. To interpret model behav- iors, we further design a linguistic acceptabil- ity task to reveal their abilities in identifying ungrammatical sentences and the position of errors. We find that fixed contextual encoders with a simple classifier trained on the predic- tion of sentence correctness are able to locate error positions. We also design a cloze test for BERT and discover that BERT captures the in- teraction between errors and specific tokens in context. Our results shed light on understand- ing the robustness and behaviors of language encoders against grammatical errors. 1 Introduction Pre-trained language encoders have achieved great success in facilitating various downstream natu- ral language processing (NLP) tasks (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b). How- ever, they usually assume training and test cor- pora are clean and it is unclear how the models behave when confronted with noisy input. Gram- matical error is an important type of noise since it naturally and frequently occurs in natural lan- guage, especially in spoken and written materials from non-native speakers. Dealing with such a noise reflects model robustness in representing lan- guage and grammatical knowledge. It would also have a positive social impact if language encoders can model texts from non-native speakers appropri- ately. Recent work on evaluating model’s behaviors against grammatical errors employs various meth- ods, including (1) manually constructing mini- mal edited pairs on specific linguistic phenom- ena (Marvin and Linzen, 2018; Goldberg, 2019; Warstadt et al., 2019a,b); (2) labeling or creating acceptability judgment resources (Linzen et al., 2016; Warstadt and Bowman, 2019; Warstadt et al., 2019a); and (3) simulating noises for a specific NLP task such as neural machine translation (Lui et al., 2018; Anastasopoulos, 2019), sentiment clas- sification (Baldwin et al., 2017). These studies either focus on specific phenomena and mainly conduct experiments on designated corpora or rely heavily on human annotations and expert knowl- edge in linguistics. In contrast, our work automat- ically simulates natural occurring data and vari- ous types of grammatical errors and systematically analyzes how these noises affect downstream ap- plications. This holds more practical significance to understand the robustness of several language encoders against grammatical errors. Specifically, we first propose an …

Content truncated to 500 words