MTEQA at WMT21 Metrics Shared Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 1024–1029 November 10–11, 2021. ©2021 Association for Computational Linguistics 1024 MTEQA at WMT21 Metrics Shared Task Mateusz Krubi´nski1, Erfan Ghadery2, Marie-Francine Moens2, and Pavel Pecina1 1Charles University, Faculty of Mathematics and Physics {krubinski,pecina}@ufal.mff.cuni.cz 2KU Leuven, Department of Computer Science {erfan.ghadery,sien.moens}@kuleuven.be Abstract In this paper, we describe our submission to the WMT 2021 Metrics Shared Task. We use the automatically-generated questions and answers to evaluate the quality of Machine Translation (MT) systems. Our submission builds upon the recently proposed MTEQA framework. Experiments on WMT20 evalua- tion datasets show that at the system-level the MTEQA metric achieves performance com- parable with other state-of-the-art solutions, while considering only a certain amount of in- formation from the whole translation. 1 Introduction The goal of automatic Machine Translation (MT) evaluation is to automatically evaluate the output quality produced by MT systems. Metrics used for this task assign a score by comparing the MT out- put to either a reference translation or to the source sentence. The main indicator that is used to assess the performance of a specific metric is the correla- tion with human judgement computed for outputs from several systems. It was recently shown that metrics based on contextualized embeddings, such as YISI (Lo, 2019) or ESIM (Mathur et al., 2019), are able to achieve better performance than the most common BLEU (Papineni et al., 2002). In this paper, we describe application of the recently proposed metric – MTEQA (Krubi´nski et al., 2021) for the task of evaluating the quality of MT outputs in the context of the WMT21 Metric task. The MTEQA1 framework is inspired by previ- ous works on evaluating abstractive summaries. It builds upon the fact that state-of-the-art (neural) MT systems tend to produce a fluent output but sometimes fail in adequacy of the translation. It leverages the recent progress in Question Genera- tion (QG) and Question Answering (QA) to formu- late and answer questions based on the MT output. 1https://github.com/ufal/MTEQA 2 Related Work 2.1 MT Evaluation Metrics that are most widely used for automatic evaluation of MT outputs produce a score by com- paring surface-level forms of hypothesis and refer- ence translation. The most common one, BLEU, is a modified version of n-gram precision calculated by averaging over different values of n with penal- ization for too short translations (brevity penalty). The recently proposed CHRF (Popovi´c, 2015) con- siders the character-level n-grams, making it possi- ble to reward partially matched tokens. Recently, various works (e.g., Lo, 2019; Mathur et al., 2019; Bawden et al., 2020) explored the usage of con- textualized word- or sentence-level embeddings to compare the numerical representations of reference and hypothesis. Such metrics enable explicit regres- sion towards the desired human-produced labels. 2.2 Question-based Evaluation Previous works examined the usage of reading com- prehension tests to measure the quality and “use- fulness” of MT systems (Tomita et al., 1993; Fuji et al., 2001; Castilho and Guerberof Arenas, 2018). Berka et al. (2011) were the first to …

Content truncated to 500 words