Abstraction not Memory: BERT and the English Article System

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 924 - 931 July 10-15, 2022 ©2022 Association for Computational Linguistics Abstraction not Memory: BERT and the English Article System Harish Tayyar Madabushi1,2, Dagmar Divjak3,4 and Petar Milin3 1 Department of Computer Science, University of Sheffield 2 School of Computer Science, University of Birmingham 3 Department of Modern Languages, University of Birmingham 4 Department of English Language and Linguistics, University of Birmingham H.TayyarMadabushi@sheffield.ac.uk (D.Divjak, P.Milin)@bham.ac.uk Abstract Article prediction is a task that has long defied accurate linguistic description. As such, this task is ideally suited to evaluate models on their ability to emulate native-speaker intuition. To this end, we compare the performance of native English speakers and pre-trained models on the task of article prediction set up as a three way choice (a/an, the, zero). Our experiments with BERT show that BERT outperforms humans on this task across all articles. In particular, BERT is far superior to humans at detecting the zero article, possibly because we insert them us- ing rules that the deep neural model can easily pick up. More interestingly, we find that BERT tends to agree more with annotators than with the corpus when inter-annotator agreement is high but switches to agreeing more with the corpus as inter-annotator agreement drops. We contend that this alignment with annotators, de- spite being trained on the corpus, suggests that BERT is not memorising article use, but cap- tures a high level generalisation of article use akin to human intuition. 1 Introduction and Motivation Pre-trained models, such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and more re- cently T5 (Raffel et al., 2020), are the state of the art across several tasks in computational linguis- tics. In addition, transformer-based models are known to have access to information as varied as part of speech information (Chrupała and Alishahi, 2019; Tenney et al., 2019b), parse trees (Hewitt and Manning, 2019), the NLP pipeline (Tenney et al., 2019a), and constructional information (Tay- yar Madabushi et al., 2020). These models tend to perform so well that, on certain tasks, they outper- form human baselines (Zhang et al., 2020). In this work, we investigate how well language models, specifically BERT Large, perform on the linguistically interesting task of article prediction. English article prediction, further discussed in Sec- tion 2, is a phenomenon that native speakers of En- glish find almost trivial. At the same time, linguists find it particularly difficult to formulate the rules that would govern article usage: article use cannot be captured by local co-occurrence but is depen- dent on the wider context and often there is no one “right” article, but multiple options are possible, albeit with slight differences in the meaning con- veyed. Grammar correction systems prior to BERT struggled to reach acceptable levels of performance on article selection (detailed in Section 2). As we will show, BERT shows performance on this task that is superior to that …

Content truncated to 500 words