Phrase-based Machine Translation is State-of-the-Art for Automatic Grammatical Error Correction

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1546–1556, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Phrase-based Machine Translation is State-of-the-Art for Automatic Grammatical Error Correction Marcin Junczys-Dowmunt and Roman Grundkiewicz Adam Mickiewicz University in Pozna´n ul. Umultowska 87, 61-614 Pozna´n, Poland {junczys,romang}@amu.edu.pl Abstract In this work, we study parameter tuning to- wards the M2 metric, the standard metric for automatic grammar error correction (GEC) tasks. After implementing M2 as a scorer in the Moses tuning framework, we investi- gate interactions of dense and sparse features, different optimizers, and tuning strategies for the CoNLL-2014 shared task. We notice er- ratic behavior when optimizing sparse feature weights with M2 and offer partial solutions. We find that a bare-bones phrase-based SMT setup with task-specific parameter-tuning out- performs all previously published results for the CoNLL-2014 test set by a large margin (46.37% M2 over previously 41.75%, by an SMT system with neural features) while be- ing trained on the same, publicly available data. Our newly introduced dense and sparse features widen that gap, and we improve the state-of-the-art to 49.49% M2. 1 Introduction Statistical machine translation (SMT), especially the phrase-based variant, is well established in the field of automatic grammatical error correction (GEC) and systems that are either pure SMT or incorporate SMT as system components occupied top positions in GEC shared tasks for different languages. With the recent paradigm shift in machine trans- lation towards neural translation models, neural encoder-decoder models are expected to appear in the field of GEC as well, and first published results (Xie et al., 2016) already look promising. As it is the case in classical bilingual machine translation research, these models should be compared against strong SMT baselines. Similarly, system combina- tions of SMT with classifier-based approaches (Ro- zovskaya and Roth, 2016) suffer from unnecessarily weak MT base systems which make it hard to assess how large the contribution of the classifier pipelines really is. In this work we provide these baselines. During our experiments, we find that a bare-bones phrase-based system outperforms the best published results on the CoNLL-2014 test set by a significant margin only due to a task-specific parameter tun- ing when being trained on the same data as previous systems. When we further investigate the influence of well-known SMT-specific features and introduce new features adapted to the problem of GEC, our fi- nal systems outperform the best reported results by 8% M2, moving the state-of-the-art results for the CoNLL-2014 test set from 41.75% M2 to 49.49%. The paper is organized as follows: section 2 describes previous work, the CoNLL-2014 shared tasks on GEC and follow-up papers. Our main con- tributions are presented in sections 3 and 4 where we investigate the interaction of parameter tuning towards the M2 metric with task-specific dense and sparse features. Especially tuning for sparse fea- tures is more challenging than initially expected, but we describe optimizer hyper-parameters that make sparse feature tuning with M2 feasible. Section 5 reports on the …

Content truncated to 500 words