KoBE: Knowledge-Based Machine Translation Evaluation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3200–3207 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3200 KoBE: Knowledge-Based Machine Translation Evaluation Zorik Gekhman Roee Aharoni Genady Beryozkin Markus Freitag Wolfgang Macherey Google Research {zorik,roeeaharoni}@google.com Abstract We propose a simple and effective method for machine translation evaluation which does not require reference translations. Our approach is based on (1) grounding the entity mentions found in each source sentence and candidate translation against a large-scale multilingual knowledge base, and (2) measuring the recall of the grounded entities found in the candi- date vs. those found in the source. Our ap- proach achieves the highest correlation with human judgements on 9 out of the 18 language pairs from the WMT19 benchmark for evalu- ation without references, which is the largest number of wins for a single evaluation method on this task. On 4 language pairs, we also achieve higher correlation with human judge- ments than BLEU. To foster further research, we release a dataset containing 1.8 million grounded entity mentions across 18 language pairs from the WMT19 metrics track data. 1 Introduction Reliable and accessible evaluation is an impor- tant catalyst for progress in machine translation (MT) and other natural language processing tasks. While human evaluation is still considered the gold- standard when done properly (L¨aubli et al., 2020), automatic evaluation is a cheaper alternative that al- lows for rapid development cycles. Today’s promi- nent automatic evaluation methods like BLEU (Pa- pineni et al., 2002) or METEOR (Banerjee and Lavie, 2005) rely on n-gram matching with refer- ence translations. While these methods are widely adopted, they have notable deficiencies: • Reference translations cover a tiny fraction of all relevant input sentences or domains, and non-professional translators yield low-quality results (Zaidan and Callison-Burch, 2011). • Different words in the candidate and reference translations that share an identical meaning The Navy of Ukraine completed the exercise in the azov sea ВМС Украины завершили учения в Азовском море The Ukrainian Navy has completed exercises in the Sea of Azov Facebook_FAIR.6937 Source online-X.0 Figure 1: Example annotations and entity matches us- ing our method. will be penalized by simple n-gram match- ing, and multiple references are rarely used to alleviate this (Qin and Specia, 2015). • Human translations have special traits (“Trans- lationese”, Koppel and Ordan, 2011) and reference-based metrics were shown to be biased to produce higher scores for transla- tionese MT outputs than for valid, alternative MT outputs (Freitag et al., 2020). • N-gram matching enables measurement of rel- ative improvements, but does not provide an interpretable quality signal (Lavie, 2010). To alleviate these issues, we propose Knowledge- Based Evaluation (KoBE), an evaluation method based on a large-scale multilingual knowledge base (KB). In our approach, we first ground each source sentence and candidate translation against the KB using entity linking (McNamee et al., 2011; Rao et al., 2013; Pappu et al., 2017; Gillick et al., 2019; Wu et al., 2019). We then measure the recall for entities found in …

Content truncated to 500 words