Just Ask! Evaluating Machine Translation by Asking and Answering Questions

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 495–506 November 10–11, 2021. ©2021 Association for Computational Linguistics 495 Just Ask! Evaluating Machine Translation by Asking and Answering Questions Mateusz Krubi´nski1, Erfan Ghadery2, Marie-Francine Moens2, and Pavel Pecina1 1Charles University Faculty of Mathematics and Physics {krubinski,pecina}@ufal.mff.cuni.cz 2KU Leuven, Department of Computer Science {erfan.ghadery,sien.moens}@kuleuven.be Abstract In this paper, we show that automatically- generated questions and answers can be used to evaluate the quality of Machine Translation systems. Building on recent work on the eval- uation of abstractive text summarization, we propose a new metric for system-level Ma- chine Translation evaluation, compare it with other state-of-the-art solutions, and show its robustness by conducting experiments for var- ious translation directions. 1 Introduction The goal of automatic Machine Translation (MT) evaluation is to automatically evaluate the output quality produced by MT systems. Metrics used for this task assign a score by comparing the MT output to either a reference translation or to the source sentence (the latter is called Quality Estimation). The main indicator that is used to assess the performance of a specific metric is the correlation with human judgement computed for outputs from several systems. It was recently shown that metrics based on contextualized embeddings, such as YISI (Lo, 2019) or ESIM (Mathur et al., 2019), are able to achieve better performance than the most widely used BLEU (Papineni et al., 2002). In this paper, we propose a new method for au- tomatic evaluation of MT systems – MTEQA1 (Machine Translation Evaluation with Question Answering), building on previous works on eval- uating abstractive summaries. We build upon the fact that state-of-the-art (neural) MT systems tend to produce a fluent output but sometimes fail in adequacy of the translation. We leverage the re- cent progress in Question Generation (QG) and Question Answering (QA) to formulate and answer human readable questions about the MT system out- put. Our experiments show that the effectiveness of the proposed metric is comparable to performance 1https://github.com/ufal/MTEQA of other automatic metrics, while considering only a certain amount of information from the whole translation. We also examine the robustness of the metric by considering several translation directions and target languages. The remainder of this paper is structured as fol- lows. In Section 2, we introduce relevant research on question-based evaluation. In Section 3, we de- scribe our metric in detail. In Section 4, we present and discuss the results of our experiments including s the influence of different human scoring methods. Section 5 presents conclusions. 2 Related Work Metrics that are most widely used for automatic evaluation of MT outputs produce a score by com- paring surface-level forms of hypothesis and refer- ence translation. The most dominant one, BLEU (Papineni et al., 2002), is a version of n-gram preci- sion calculated by averaging over different values of n with penalization for overly short translations (brevity penalty). Another one, CHRF (Popovi´c, 2015), considers the character-level n-grams, mak- ing it possible to reward partial token matches. The standardised implementation provided in …

Content truncated to 500 words