Word Embedding-Based Automatic MT Evaluation Metric using Word Position Information

Proceedings of NAACL-HLT 2019, pages 1874–1883 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1874 Word Embedding-Based Automatic MT Evaluation Metric using Word Position Information Hiroshi Echizen’ya Department of Life Science and Technology Hokkai-Gakuen University, Japan echi@lst.hokkai-s-u.ac.jp Kenji Araki Graduate School of Information Science and Technology Hokkaido University, Japan araki@ist.hokudai.ac.jp Eduard Hovy Language Technologies Institute Carnegie Mellon University, USA hovy@cmu.edu Abstract We propose a new automatic evaluation met- ric for machine translation. Our proposed met- ric is obtained by adjusting the Earth Mover’s Distance (EMD) to the evaluation task. The EMD measure is used to obtain the distance between two probability distributions con- sisting of some signatures having a feature and a weight. We use word embeddings, sentence-level tf · idf, and cosine similarity between two word embeddings, respectively, as the features, weight, and the distance be- tween two features. Results show that our proposed metric can evaluate machine trans- lation based on word meaning. Moreover, for distance, cosine similarity and word po- sition information are used to address word- order differences. We designate this met- ric as Word Embedding-based automatic MT evaluation using Word Position Information (WE WPI). A meta-evaluation using WMT16 metrics shared task set indicates that our WE WPI achieves the highest correlation with human judgment among several representative metrics. 1 Introduction Recent advances in neural machine translation (NMT) (Sutskever et al., 2014; Luong et al., 2015) are remarkable. Results based on hu- man evaluation have demonstrated that NMT out- performs statistical machine translations signifi- cantly (Chiang, 2005; Tufis¸ and Ceaus¸u, 2009). The NMT achieved especially high performance in terms of fluency. However, it tends to gener- ate more omission errors than statistical machine translations generate. Unfortunately, it is diffi- cult for automatic evaluation metrics to evaluate outputs with omission errors because those errors are not included as non-match words between the translation and reference. For such cases, the word embedding-based automatic MT evaluation met- ric, which is based on word position information, is effective. Various automatic evaluation metrics have been proposed for machine translation, but none is suf- ficient for NMT. Actually, BLEU (Papineni et al., 2002) is the representative metric based on n- gram matching. Unfortunately, because it is a surface-level metric, it is difficult to address word meaning during evaluation for MT out- puts. The word-embedding-based distance mea- sure for document (Kusner et al., 2016) and the word-alignment-based automatic evaluation met- ric using word embedding (Matsuo et al., 2017) are effective to address word meanings. Never- theless, they can only ineffectively accommodate word order differences between the translation and reference. Given those circumstances, a new metric with word embedding-based automatic MT evalua- tion metric using word position information is proposed in which the evaluation score is ob- tained by adjusting the Earth Mover’s Distance (EMD) (Rubner et al., 1998, 2000) to the eval- uation task. The EMD measure represents the distance between two probability distributions. Moreover, the EMD distance is obtained based on a signature consisting of the feature …

Content truncated to 500 words