Adaptive HTER Estimation for Document-Specific MT Post-Editing

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 861–870, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Adaptive HTER Estimation for Document-Specific MT Post-Editing Fei Huang ∗ Facebook Inc. Menlo Park, CA feihuang@fb.com Jian-Ming Xu Abraham Ittycheriah IBM T.J. Watson Research Center Yorktown Heights, NY {jianxu, abei, roukos}@us.ibm.com Salim Roukos Abstract We present an adaptive translation qual- ity estimation (QE) method to predict the human-targeted translation error rate (HTER) for a document-specific machine translation model. We first introduce fea- tures derived internal to the translation de- coding process as well as externally from the source sentence analysis. We show the effectiveness of such features in both classification and regression of MT qual- ity. By dynamically training the QE model for the document-specific MT model, we are able to achieve consistency and pre- diction quality across multiple documents, demonstrated by the higher correlation co- efficient and F-scores in finding Good sen- tences. Additionally, the proposed method is applied to IBM English-to-Japanese MT post editing field study and we observe strong correlation with human preference, with a 10% increase in human translators’ productivity. 1 Introduction Machine translation (MT) systems suffer from an inconsistent and unstable translation quality. De- pending on the difficulty of the input sentences (sentence length, OOV words, complex sentence structures and the coverage of the MT system’s training data), some translation outputs can be per- fect, while others are ungrammatical, missing im- portant words or even totally garbled. As a result, users do not know whether they can trust the trans- lation output unless they spend time to analyze ∗This work was done when the author was with IBM Re- search. the MT output. This shortcoming is one of the main obstacles for the adoption of MT systems, especially in machine assisted human translation: MT post-editing, where human translators have an option to edit MT proposals or translate from scratch. It has been observed that human trans- lators often discard MT proposals even if some are very accurate. If MT proposals are used prop- erly, post-editing can increase translators produc- tivity and lead to significant cost savings. There- fore, it is beneficial to provide MT confidence es- timation, to help the translators to decide whether to accept MT proposals, making minor modifica- tions on MT proposals when the quality is high or translating from scratching when the quality is low. This will save the time of reading and parsing low quality MT and improve user experience. In this paper we propose an adaptive qual- ity estimation that predicts sentence-level human- targeted translation error rate (HTER) (Snover et al., 2006) for a document-specific MT post-editing system. HTER is an ideal quality measurement for MT post editing since the reference is ob- tained from human correction of the MT output. Document-specific MT model is an MT model that is specifically built for the given input document. It is demonstrated in (Roukos et al., 2012) that document-specific MT models significantly im- prove the translation …

Content truncated to 500 words