A Closer Look at Linguistic Knowledge in Masked Language Models: The Case of Relative Clauses in American English

Proceedings of the 28th International Conference on Computational Linguistics, pages 771–787 Barcelona, Spain (Online), December 8-13, 2020 771 A Closer Look at Linguistic Knowledge in Masked Language Models: The Case of Relative Clauses in American English Marius Mosbach∗, Stefania Degaetano-Ortlieb∗, Marie-Pauline Krielke, Badr M. Abdullah, Dietrich Klakow Department of Language Science and Technology, Saarland University, Germany {mmosbach,babdullah,dklakow}@lsv.uni-saarland.de s.degaetano@mx.uni-saarland.de mariepauline.krielke@uni-saarland.de Abstract Transformer-based language models achieve high performance on various tasks, but we still lack understanding of the kind of linguistic knowledge they learn and rely on. We evaluate three models (BERT, RoBERTa, and ALBERT), testing their grammatical and semantic knowledge by sentence-level probing, diagnostic cases, and masked prediction tasks. We focus on relative clauses (in American English) as a complex phenomenon needing contextual information and an- tecedent identification to be resolved. Based on a naturalistic dataset, probing shows that all three models indeed capture linguistic knowledge about grammaticality, achieving high performance. Evaluation on diagnostic cases and masked prediction tasks considering fine-grained linguis- tic knowledge, however, shows pronounced model-specific weaknesses especially on semantic knowledge, strongly impacting models’ performance. Our results highlight the importance of (a) model comparison in evaluation task and (b) building up claims of model performance and the linguistic knowledge they capture beyond purely probing-based evaluations. 1 Introduction Endeavors to better understand transformer-based masked language models (MLMs), such as BERT, are ever growing since their introduction in 2017 (cf. Rogers et al. (2020) for an overview). While the BERTology movement has enhanced our knowledge on the reasons behind BERT’s performance in vari- ous ways, still plenty remains unanswered. Less well studied and challenging are linguistic phenomena, where, besides contextual information, identification of an antecedent is needed, such as relative clauses (RCs). Kim et al. (2019), e.g., analyzed BERT’s comprehension of function words, showing how rela- tivizers and prepositions are quite challenging for BERT. Similarly, Warstadt and Bowman (2019) find RCs to be difficult for BERT in the CoLA acceptability tasks. In this paper, we focus on RCs in Ameri- can English to further enhance our understanding of the grammatical and semantic knowledge captured by pre-trained MLMs, evaluating three models: BERT, RoBERTa, and ALBERT. For our analysis, we train probing classifiers, consider each models’ performance on diagnostic cases, and test predictions in a masked language modeling task on selected semantic and grammatical constraints of RCs. RCs are clausal post-modifiers specifying a preceding noun phrase (antecedent) and are introduced by a relativizer (e.g., which). Extensive corpus research (Biber et al., 1999) found that the overall most common relativizers are that, which, and who. The relativizer occupies the subject or object position in a sentence (see examples (1-a) and (1-b)). In subject RCs, the relativizer is obligatory (Huddleston and Pullum, 2002, 1055), while in object position omission is licensed (e.g., zero in example (1-b)). (1) a. Children who eat vegetables are likely to be healthy. (subject relativizer, relativizer is oblig- atory) b. This is the dress [that/which/zero] I brought yesterday. (object relativizer, omission possible) ∗ equal contribution. This work is licensed under a …

Content truncated to 500 words