Identifying Fluently Inadequate Output in Neural and Statistical Machine Translation

Identifying Fluently Inadequate Output in Neural and Statistical Machine Translation Marianna J. Martindale† Marine Carpuat‡ Kevin Duh◦ Paul McNamee◦ †iSchool, ‡Dept. of Computer Science, University of Maryland, College Park, USA ◦HLTCOE, Johns Hopkins University, Baltimore, USA mmartind@umiacs.umd.edu, marine@cs.umd.edu kevinduh@cs.jhu.edu, mcnamee@jhu.edu Abstract With the impressive fluency of modern ma- chine translation output, systems may pro- duce output that is fluent but not adequate (fluently inadequate). We seek to identify these errors and quantify their frequency in MT output of varying quality. To that end, we introduce a method for automat- ically predicting whether translated seg- ments are fluently inadequate by predicting fluency using grammaticality scores and predicting adequacy by augmenting sen- tence BLEU with a novel Bag-of-Vectors Sentence Similarity (BVSS). We then ap- ply this technique to analyze the outputs of statistical and neural systems for six lan- guage pairs with different levels of trans- lation quality. We find that neural mod- els are consistently more prone to this type of error than traditional statistical models. However, improving the overall quality of the MT system such as through domain adaptation reduces these errors. 1 Introduction Recent work has shown that well-trained, in- domain neural machine translation (NMT) systems can produce translations that, at the sentence level, are rated on par with human reference transla- tions (Hassan Awadalla et al., 2018). Part of this success comes from the impressive improvements in fluency of NMT output compared to previous MT paradigms (Bentivogli et al., 2016; Toral and Sánchez-Cartagena, 2017; Koehn and Knowles, c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. 2017). However, NMT has also been shown to sometimes produce output that is low adequacy and even unrelated to the input–particularly when not trained on sufficient in-domain data (Koehn and Knowles, 2017). Because of NMT’s uncanny ability to produce fluent output, these translations may not just be inadequate but fluently inade- quate. The fluency of fluently inadequate trans- lations may mislead users into trusting the content based on fluency alone–particularly in the context of other fluent and adequate translations (Martin- dale and Carpuat, 2018). Mitigating the effects of fluently inadequate translations first requires understanding the scale of the problem and what situations are likely to generate these errors. The general success and high system level quality of NMT suggests that flu- ently inadequate translations are rare, but we can- not say how rare without a means of automatically identifying potentially fluently inadequate transla- tions in large collections of MT output. In this work, we propose a method to automati- cally detect fluently inadequate translations based on the underlying characteristics of fluency and adequacy. We view fluently inadequate transla- tions as translations that are fluent, well-formed sentences that could have been written by a hu- man, and that do not preserve the meaning of the reference. In practice, given a reference trans- lation r and MT hypothesis h, we consider h to be fluently inadequate if fluency(h) > τf and adequacy(h, r) …

Content truncated to 500 words