The Trouble with SMT Consistency

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 442–449, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics The Trouble with SMT Consistency Marine Carpuat and Michel Simard National Research Council Canada 283 Alexandre-Tach´e Boulevard Building CRTL, Room F-2007 Gatineau (Qu´ebec) J8X 3X7 Firstname.Lastname@nrc.ca Abstract SMT typically models translation at the sen- tence level, ignoring wider document context. Does this hurt the consistency of translated documents? Using a phrase-based SMT sys- tem in various data conditions, we show that SMT translates documents remarkably con- sistently, even without document knowledge. Nevertheless, translation inconsistencies often indicate translation errors. However, unlike in human translation, these errors are rarely due to terminology inconsistency. They are more often symptoms of deeper issues with SMT models instead. 1 Introduction While Statistical Machine Translation (SMT) mod- els translation at the sentence level (Brown et al., 1993), human translators work on larger translation units. This is partly motivated by the importance of producing consistent translations at the document level. Consistency checking is part of the quality as- surance process, and complying with the terminol- ogy requirements of each task or client is crucial. In fact, many automatic tools have been proposed to assist humans in this important task (Itagaki et al., 2007; Dagan and Church, 1994, among others). This suggests that wider document-level context information might benefit SMT models. However, we do not have a clear picture of the impact of sentence-based SMT on the translation of full doc- uments. From a quality standpoint, it seems safe to assume that translation consistency is as desirable for SMT as for human translations. However, con- sistency needs to be balanced with other quality re- quirements. For instance, strict consistency might result in awkward repetitions that make translations less fluent. From a translation modeling standpoint, while typical SMT systems do not explicitly enforce translation consistency, they can learn lexical choice preferences from training data in the right domain. In this paper, we attempt to get a better under- standing of SMT consistency. We conduct an em- pirical analysis using a phrase-based SMT system in a variety of experimental settings, focusing on two simple, yet understudied, questions. Is SMT output consistent at the document level? Do inconsistencies indicate translation errors? We will see that SMT consistency issues are quite different from consistency issues in human transla- tions. In fact, while inconsistency errors in SMT output might be particularly obvious to the human eye, SMT is globally about as consistent as human translations. Furthermore, high translation consis- tency does not guarantee quality: weaker SMT sys- tems trained on less data translate more consistently than stronger larger systems. Yet, inconsistent trans- lations often indicate translation errors, possibly be- cause words and phrases that translate inconsistently are the hardest to translate. After discussing related work on consistency and document modeling for SMT (Section 2), we de- scribe our corpora in Section 3 and our general methodology in Section 4. In Section 5, we dis- cuss the results of an automatic analysis of …

Content truncated to 500 words