Building a State-of-the-Art Grammatical Error Correction System

Building a State-of-the-Art Grammatical Error Correction System Alla Rozovskaya Center for Computational Learning Systems Columbia University New York, NY 10115 alla@ccls.columbia.edu Dan Roth Department of Computer Science University of Illinois Urbana, IL 61801 danr@illinois.edu Abstract This paper identifies and examines the key principles underlying building a state-of-the- art grammatical error correction system. We do this by analyzing the Illinois system that placed first among seventeen teams in the re- cent CoNLL-2013 shared task on grammatical error correction. The system focuses on five different types of errors common among non-native English writers. We describe four design principles that are relevant for correcting all of these er- rors, analyze the system along these dimen- sions, and show how each of these dimensions contributes to the performance. 1 Introduction The field of text correction has seen an increased interest in the past several years, with a focus on correcting grammatical errors made by English as a Second Language (ESL) learners. Three competi- tions devoted to error correction for non-native writ- ers took place recently: HOO-2011 (Dale and Kil- garriff, 2011), HOO-2012 (Dale et al., 2012), and the CoNLL-2013 shared task (Ng et al., 2013). The most recent and most prominent among these, the CoNLL-2013 shared task, covers several common ESL errors, including article and preposition usage mistakes, mistakes in noun number, and various verb errors, as illustrated in Fig. 1.1 Seventeen teams that 1The CoNLL-2014 shared task that completed at the time of writing this paper was an extension of the CoNLL-2013 com- petition (Ng et al., 2014) but addressed all types of errors. The Illinois-Columbia submission, a slightly extended version of the Nowadays *phone/phones *has/have many functionalities, *included/including *∅/a camera and *∅/a Wi-Fi receiver. Figure 1: Examples of representative ESL errors. participated in the task developed a wide array of ap- proaches that include discriminative classifiers, lan- guage models, statistical machine-translation sys- tems, and rule-based modules. Many of the systems also made use of linguistic resources such as addi- tional annotated learner corpora, and defined high- level features that take into account syntactic and se- mantic knowledge. Even though the systems incorporated similar re- sources, the scores varied widely. The top system, from the University of Illinois, obtained an F1 score of 31.202, while the second team scored 25.01 and the median result was 8.48 points.3 These results suggest that there is not enough understanding of what works best and what elements are essential for building a state-of-the-art error correction system. In this paper, we identify key principles for build- ing a robust grammatical error correction system and show their importance in the context of the shared task. We do this by analyzing the Illinois system and evaluating it along several dimensions: choice Illinois CoNLL-2013 system, ranked at the top. For a descrip- tion of the Illinois-Columbia submission, we refer the reader to Rozovskaya et al. (2014a). 2The state-of-the-art performance of the Illinois system dis- cussed here is with respect to individual components for differ- ent errors. Improvements in Rozovskaya and Roth …

Content truncated to 500 words