Neural Machine Translation of Text from Non-Native Speakers

Proceedings of NAACL-HLT 2019, pages 3070–3080 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3070 Neural Machine Translation of Text from Non-Native Speakers Antonios Anastasopoulos†,1 Alison Lui†,2 Toan Q. Nguyen2 David Chiang2 1Language Technologies Institute, Carnegie Mellon University 2Department of Computer Science and Engineering, University of Notre Dame aanastas@cs.cmu.edu {alui,tnguye28,dchiang}@nd.edu Abstract Neural Machine Translation (NMT) systems are known to degrade when confronted with noisy data, especially when the system is trained only on clean data. In this paper, we show that augmenting training data with sen- tences containing artificially-introduced gram- matical errors can make the system more ro- bust to such errors. In combination with an au- tomatic grammar error correction system, we can recover 1.0 BLEU out of 2.4 BLEU lost due to grammatical errors. We also present a set of Spanish translations of the JFLEG gram- mar error correction corpus, which allows for testing NMT robustness to real grammatical errors. 1 Introduction Neural Machine Translation (NMT) is undeniably a success story: public benchmarks (Bojar et al., 2016) are dominated by neural systems, and neu- ral approaches are the de facto option for industrial systems (Wu et al., 2016; Hassan Awadalla et al., 2018; Crego et al., 2016; Hieber et al., 2018). Even under low-resource conditions, neural mod- els were recently shown to outperform traditional statistical approaches (Nguyen and Chiang, 2018). However, there are still several shortcomings of NMT that need to be addressed: a (non- exhaustive) list of six challenges is discussed by Koehn and Knowles (2017), including out- of-domain testing, rare word handling, the wide- beam problem, and the large amount of data needed for learning. An additional challenge is ro- bustness to noise, both during training and at in- ference time. In this paper, we study the effect of a specific type of noise in NMT: grammatical errors. We pri- marily focus on errors that are made by non-native †Equal contribution. Work performed at the University of Notre Dame. source-language speakers (as opposed to dialectal language, SMS or Twitter language). Not only is this linguistically important, but we believe that it would potentially have great social impact. Our contributions are three-fold. First, we con- firm that NMT is vulnerable to source-side noise when trained on clean data, losing up to 3.6 BLEU on our test set. This is consistent with previous work, yet orthogonal to it, since we use more re- alistic noise for our experiments. Second, we ex- plore training methods that can deal with noise, and show that including noisy synthetic data in the training data makes NMT more robust to handling similar types of errors in test data. Combining this simple method with an automatic grammar cor- rection system, we find that we can recover 1.5 BLEU. Third, we release Spanish translations of the JFLEG corpus,1 a standard benchmark for En- glish Grammar Error Correction (GEC) systems. We also release all other data and code used in this paper. Our additional annotations on both the JFLEG corpus and …

Content truncated to 500 words