Predicting Machine Translation Adequacy with Document Embeddings

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 402–410, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Predicting Machine Translation Adequacy with Document Embeddings Mihaela Vela Saarland University Saarbr¨ucken, Germany m.vela@mx.uni-saarland.de Liling Tan Saarland University Saarbr¨ucken, Germany liling.tan@uni-saarland.de Abstract This paper describes USAAR’s submis- sion to the the metrics shared task of the Workshop on Statistical Machine Transla- tion (WMT) in 2015. The goal of our sub- mission is to take advantage of the seman- tic overlap between hypothesis and ref- erence translation for predicting MT out- put adequacy using language independent document embeddings. The approach pre- sented here is learning a Bayesian Ridge Regressor using document skip-gram em- beddings in order to automatically evalu- ate Machine Translation (MT) output by predicting semantic adequacy scores. The evaluation of our submission – measured by the correlation with human judgements – shows promising results on system-level scores. 1 Introduction Translation is becoming an utility in everyday life. The increased availability of real-time machine translation services relying on Statistical Machine Translation (SMT) allows users who do not un- derstand the language of the source text to quickly gist the text and understand its general meaning. For these users, accurate meaning of translated words is more important than the fluency of the translated sentence. However, SMT suffers from poor lexical choices. Fluent but inadequate translations are commonly produced due to the strong bias to- wards the language model component that prefers consecutive words based on the data that the sys- tem is trained on. Current state of art MT evaluation metrics are generally able to identify problems with grammat- icality of the translation but less evidently accu- racy of translated semantics, e.g. incorrect trans- lation of ambiguous words or wrong assignment of semantic roles. In the example below, the ideal Machine Translation (MT) evaluation metric should appropriately penalise poor lexical choice, such as braked, and reward or at least allow lee- way for semantically similar translations, such as external trade. Source (DE): Auch der Auenhandel bremste die Konjunktur. Phrase-based MT: The foreign trade braked the economy. Neural MT: External trade also slowed the economy. Reference (EN): Foreign goods trade had slowed, too. The German word bremste is commonly used as braked in the context of driving, but the appro- priate translation should have been slowed in the example mentioned above. Although the phrase external trade differs from foreign goods trade in the reference sentence, it should be considered as an acceptable translation. We propose a semantically grounded, language independent approach using Semantic Textual Similarity (STS) to evaluate the adequacy of the machine translation outputs with respect to their reference translations. The remainder of this paper is structured as fol- lows. Section 2 gives an overview of the related work in the field of MT evaluation. Section 3 presents the approach behind the USAAR submis- sion to the metrics shared task. In Section 4 we present the data and experiments for this submis- sion. Section 5 covers the evaluation of our …

Content truncated to 500 words