A Simple and Robust Approach to Detecting Subject-Verb Agreement Errors

Proceedings of NAACL-HLT 2019, pages 2418–2427 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2418 A Simple and Robust Approach to Detecting Subject–Verb Agreement Errors Simon Flachs1,2, Ophélie Lacroix1, Marek Rei3, Helen Yannakoudakis3, Anders Søgaard2 1 Siteimprove, Denmark 2 CoAStaL DIKU, Department of Computer Science, University of Copenhagen, Denmark 3 The ALTA Institute, Dept. of CS & Technology, University of Cambridge, United Kingdom {sfl, ola}@siteimprove.com, {marek.rei, helen.yannakoudakis}@cl.cam.ac.uk, soegaard@di.ku.dk Abstract While rule-based detection of subject–verb agreement (SVA) errors is sensitive to syntac- tic parsing errors and irregularities and excep- tions to the main rules, neural sequential la- belers have a tendency to overfit their training data. We observe that rule-based error genera- tion is less sensitive to syntactic parsing errors and irregularities than error detection and ex- plore a simple, yet efficient approach to getting the best of both worlds: We train neural sequ- ential labelers on the combination of large vo- lumes of silver standard data, obtained through rule-based error generation, and gold standard data. We show that our simple protocol leads to more robust detection of SVA errors on both in-domain and out-of-domain data, as well as in the context of other errors and long-distance dependencies; and across four standard bench- marks, the induced model on average achieves a new state of the art. 1 Introduction Grammatical Error Detection. Grammatical Error Detection (GED, Leacock et al., 2010) is the task of detecting grammatical errors in text. It is used in various real-world applications, such as writing assistance tools, self-assessment fra- meworks and language tutoring systems, facili- tating incremental and/or exploratory editing of one’s writing. Accurate error detection systems also have potential applications for language ge- neration and machine translation systems, guiding automatically generated output towards gramma- tically correct sequences. The problem of detecting subject–verb agre- ement (SVA) errors is an important subtask of GED. In this work, we focus on detecting subject– verb agreement errors in the English as a Second Language (ESL) domain. Most SVA errors occur at the third-person present tense when determining whether the subject describes a singular or a plu- ral concept. The following examples demonstrate subject–verb agreement errors (bold): (1) a. *They all knows where the conference is. b. *The Hotel are very close to Town Hall. The task can be formulated as a sequence labe- ling problem, with the goal of labeling subject– verb pairs as being in agreement or not. Approaches. Sequence labeling problems in NLP, including GED and the subtask of iden- tifying SVA errors, have, in recent years, been handled with Recurrent Neural Networks (RNNs) trained on large amounts of data (Rei and Yan- nakoudakis, 2016, 2017). However, most publi- cly available datasets for GED are relatively small, making it difficult to learn a general grammar re- presentation and potentially leading to over-fitting. Previous work has also shown that neural langu- age models with a similar architecture have diffi- culty learning subject–verb agreement patterns in the presence of agreement attractors (Linzen et al., 2016). Rule-based …

Content truncated to 500 words