Probing for targeted syntactic knowledge through grammatical error detection

Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 360 - 373 December 7-8, 2022 ©2022 Association for Computational Linguistics Probing for targeted syntactic knowledge through grammatical error detection Christopher Davis† Christopher Bryant† Andrew Caines† Marek Rei‡† Paula Buttery† †ALTA Institute, Department of Computer Science & Technology, University of Cambridge, U.K. ‡Imperial College London, U.K. †{ccd38,cjb255,apc38,pjb48}@cam.ac.uk ‡marek.rei@imperial.ac.uk Abstract Targeted studies testing knowledge of subject- verb agreement (SVA) indicate that pre-trained language models encode syntactic informa- tion. We assert that if models robustly encode subject-verb agreement, they should be able to identify when agreement is correct and when it is incorrect. To that end, we propose gram- matical error detection as a diagnostic probe to evaluate token-level contextual representa- tions for their knowledge of SVA. We eval- uate contextual representations at each layer from five pre-trained English language models: BERT, XLNET, GPT-2, ROBERTA, and ELEC- TRA. We leverage public annotated training data from both English second language learn- ers and Wikipedia edits, and report results on manually crafted stimuli for subject-verb agree- ment. We find that masked language models linearly encode information relevant to the de- tection of SVA errors, while the autoregressive models perform on par with our baseline. How- ever, we also observe a divergence in perfor- mance when probes are trained on different training sets, and when they are evaluated on different syntactic constructions, suggesting the information pertaining to SVA error detection is not robustly encoded. 1 Introduction Recent work investigates whether linguistic infor- mation is encoded in pre-trained transformer-based language models (Peters et al., 2018; Devlin et al., 2019). Research using diagnostic methods (Shi et al., 2016; Alain and Bengio, 2017; Adi et al., 2017; Conneau et al., 2018; Hupkes et al., 2018) indicates models encode syntax via experiments targeting, for example, part-of-speech and depen- dency labelling (Tenney et al., 2019; Jawahar et al., 2019; Hewitt and Manning, 2019), while targeted syntactic evaluation studies show models encode a large amount of hierarchical syntactic informa- tion in tests for subject-verb agreement (Linzen et al., 2016; Marvin and Linzen, 2018; Goldberg, 2019). Although previous research has covered a large number of probing tasks (Tenney et al., 2019; Liu et al., 2019a), no one has yet fully explored grammatical error detection (GED) as a probe. We assert that the ability to detect ungrammatical to- kens serves as a complementary evaluation to as- sess linguistic competence. GED is a natural and complex NLP task that as- sesses a model’s ability to detect which tokens in a sentence are grammatically incorrect. Ungrammat- ical tokens may be categorised within a taxonomy1 comprising three operational categories (replace- ment, unnecessary, and missing) and twenty-five categories based on parts-of-speech. For example: (1) [Replacement subject-verb agreement] The train are a good option for long trips. (2) [Replacement pronoun] Everybody must have free time for yourself. (3) [Missing determiner] The birth of [a] new star. (4) [Unnecessary preposition] Public trans- port means travelling around [...] by using trains, buses, and planes. To do well in …

Content truncated to 500 words