Integrating Meaning into Quality Evaluation of Machine Translation

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 210–219, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Integrating Meaning into Quality Evaluation of Machine Translation Osman Bas¸kaya1, Eray Yıldız1, Doruk Tunao˘glu1, M. Tolga Eren1, and A. Seza Do˘gru¨oz2 1Huawei Turkey Research and Development Center, Istanbul, Turkey {osbaskaya,doruktuna,tolgaeren}@gmail.com, eray.yildiz@huawei.com 2Independent Researcher a.s.dogruoz@gmail.com Abstract Machine translation (MT) quality is eval- uated through comparisons between MT outputs and the human translations (HT). Traditionally, this evaluation relies on form related features (e.g. lexicon and syn- tax) and ignores the transfer of meaning reflected in HT outputs. Instead, we eval- uate the quality of MT outputs through meaning related features (e.g. polarity, subjectivity) with two experiments. In the first experiment, the meaning related features are compared to human rankings individually. In the second experiment, combinations of meaning related features and other quality metrics are utilized to predict the same human rankings. The re- sults of our experiments confirm the ben- efit of these features in predicting hu- man evaluation of translation quality in ad- dition to traditional metrics which focus mainly on form. 1 Introduction Machine translation (MT) systems translate large chunks of data automatically across languages. Although these systems may achieve high level accuracies using form related features (e.g. lexi- cal and syntactic), they often fail to carry over the meaning embracing the form. Example (1) high- lights the meaning difference between a Human Translation (HT) and an MT output for the same source sentence: Example (1) HT: “Your feet’s too big.”1 MT: “Your feet is too great.”2 Although the form is often preserved, MT out- puts may sound “strange” or “different” in com- parison to HT ones due to the loss of meaning. Therefore, human translators generally enrich the text with the appropriate tone, style and sentiments during translation. Current quality evaluation met- rics like BLEU (Papineni et al., 2002) and ME- TEOR (Lavie and Agarwal, 2007) are based on form related features and do not directly consider the transfer of meaning (e.g. sentiment and style) in MT. Some of these metrics check for synonyms and paraphrases but this approach is still limited to the coverage of the corresponding pair tables. In other words, these metrics do not explicitly evalu- ate the transfer of meaning in MT. Our main goals are: • to find out whether the transfer of meaning related features (e.g. sentiment and style) in MT influences the human judgment of trans- lation quality. • to compare meaning and form related fea- tures for quality evaluation of MT. • to measure whether meaning and form re- lated features can be combined to improve the performance of existing MT quality eval- uation metrics. 1WMT’15 Finnish to English test set, reference transla- tion, segment id:440 2WMT’15 Finnish to English test set, translated by sys- tem: UoS.4059, segment id:440 210 By using publicly available parallel corpora (Tenth Workshop on Statistical Machine Transla- tion (WMT15)), we achieve our goals with two …

Content truncated to 500 words