Adapting Sequence Models for Sentence Correction

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2807–2813 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Adapting Sequence Models for Sentence Correction Allen Schmaltz* Yoon Kim Alexander M. Rush Stuart M. Shieber Harvard University {schmaltz@fas,yoonkim@seas,srush@seas,shieber@seas}.harvard.edu *Part of this work was completed while as an intern at Rakuten. Abstract In a controlled experiment of sequence-to- sequence approaches for the task of sen- tence correction, we find that character- based models are generally more effec- tive than word-based models and models that encode subword information via con- volutions, and that modeling the output data as a series of diffs improves effec- tiveness over standard approaches. Our strongest sequence-to-sequence model im- proves over our strongest phrase-based statistical machine translation model, with access to the same data, by 6 M2 (0.5 GLEU) points. Additionally, in the data environment of the standard CoNLL-2014 setup, we demonstrate that modeling (and tuning against) diffs yields similar or better M2 scores with simpler models and/or significantly less data than previous sequence-to-sequence approaches. 1 Introduction The task of sentence correction is to convert a natural language sentence that may or may not have errors into a corrected version. The task is envisioned as a component of a learning tool or writing-assistant, and has seen increased interest since 2011 driven by a series of shared tasks (Dale and Kilgarriff, 2011; Dale et al., 2012; Ng et al., 2013, 2014). Most recent work on language correction has focused on the data provided by the CoNLL-2014 shared task (Ng et al., 2014), a set of corrected es- says by second-language learners. The CoNLL- 2014 data consists of only around 60,000 sen- tences, and as such, competitive systems have made use of large amounts of corrected text with- out annotations, and in some cases lower-quality crowd-annotated data, in addition to the shared data. In this data environment, it has been sug- gested that statistical phrase-based machine trans- lation (MT) with task-specific features is the state-of-the-art for the task (Junczys-Dowmunt and Grundkiewicz, 2016), outperforming word- and character-based sequence-to-sequence models (Yuan and Briscoe, 2016; Xie et al., 2016; Ji et al., 2017), phrase-based systems with neural features (Chollampatt et al., 2016b,a), re-ranking output from phrase-based systems (Hoang et al., 2016), and combining phrase-based systems with classi- fiers trained for hand-picked subsets of errors (Ro- zovskaya and Roth, 2016). We revisit the comparison across translation ap- proaches for the correction task in light of the Au- tomated Evaluation of Scientific Writing (AESW) 2016 dataset, a correction dataset containing over 1 million sentences, holding constant the training data across approaches. The dataset was previ- ously proposed for the distinct binary classifica- tion task of grammatical error identification. Experiments demonstrate that pure character- level sequence-to-sequence models are more ef- fective on AESW than word-based models and models that encode subword information via con- volutions over characters, and that representing the output data as a series of diffs significantly in- creases effectiveness on this task. Our strongest character-level model achieves statistically …

Content truncated to 500 words