Automatic Machine Translation Evaluation using Source Language Inputs and Cross-lingual Language Model

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3553–3558 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3553 Automatic Machine Translation Evaluation using Source Language Inputs and Cross-lingual Language Model Kosuke Takahashi1, Katsuhito Sudoh1,,2, Satoshi Nakamura1 1 Nara Institute of Science and Technology 2 PRESTO, Japan Science and Technology Agency {takahashi.kosuke.th0, sudoh, s-nakamura}@is.naist.jp Abstract We propose an automatic evaluation method of machine translation that uses source language sentences regarded as additional pseudo refer- ences. The proposed method evaluates a trans- lation hypothesis in a regression model. The model takes the paired source, reference, and hypothesis sentence all together as an input. A pretrained large scale cross-lingual language model encodes the input to sentence-pair vec- tors, and the model predicts a human evalua- tion score with those vectors. Our experiments show that our proposed method using Cross- lingual Language Model (XLM) trained with a translation language modeling (TLM) objec- tive achieves a higher correlation with human judgments than a baseline method that uses only hypothesis and reference sentences. Ad- ditionally, using source sentences in our pro- posed method is confirmed to improve the eval- uation performance. 1 Introduction Automatic machine translation evaluation (MTE) has been studied to substitute human evaluation in machine translation development because it is low-cost, handy, and stable to use. Popular auto- matic MTE metrics such as BLEU (Papineni et al., 2002) calculate the evaluation score based on a surface-level similarity of a paired 1-to-1 reference and translated hypothesis sentences. BLEU partic- ularly evaluates the sentence similarity with the n- gram word matching rate between a reference and hypothesis. However, the evaluation score drops when a reference and hypothesis are dissimilar in the surface even if they share the same meaning. To counter this problem, METEOR (Banerjee and Lavie, 2005) is proposed to mitigate the word matching of synonyms with a synonym dictionary. Yet still, with mitigation of word matching, surface- level similarity cannot fully compensate for seman- tics, thus word representation instead of word sym- bols is used in Word Mover’s Distance (Kusner et al., 2015) and bleu2vec (T¨attar and Fishel, 2017). Besides, sentence representation is known to be an efficient feature instead of word representation because sentence vectors can represent more global meanings. RUSE (Shimanaka et al., 2018) and BERT (Devlin et al., 2019) based MTE, BERT re- gressor (Shimanaka et al., 2019), utilized sentence representation and performed well on WMT17 Met- ric Shared Task (Bojar et al., 2017). The metrics mentioned above compare a hypothesis translation to a reference. However, a reference translation represents only one possible translation and those MTE metrics are unlikely to correctly evaluate all candidates that share the same meanings of the reference or have fatally different meanings due to a few translation errors. This problem can be mitigated by the use of multiple reference transla- tions as argued by Dreyer and Marcu (2012) and Qin and Specia (2015), but preparing such multiple references is costly. Hereby, we propose a method to incorporate source …

Content truncated to 500 words