Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3105–3117 August 1–6, 2021. ©2021 Association for Computational Linguistics 3105 Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort Vˆania Mendonc¸a1,2, Ricardo Rei1,2,3, Lu´ısa Coheur1,2, Alberto Sardinha1,2, Ana L´ucia Santos4,5 1 INESC-ID Lisboa, Portugal 2 Instituto Superior T´ecnico, Universidade de Lisboa, Portugal 3 Unbabel AI, Lisboa, Portugal 4 Centro de Lingu´ıstica da Universidade de Lisboa, Portugal 5 Faculdade de Letras da Universidade de Lisboa, Portugal {vania.mendonca, luisa.coheur, jose.alberto.sardinha}@tecnico.ulisboa.pt, ricardo.rei@unbabel.com, als@letras.ulisboa.pt Abstract In Machine Translation, assessing the quality of a large amount of automatic translations can be challenging. Automatic metrics are not re- liable when it comes to high performing sys- tems. In addition, resorting to human evalua- tors can be expensive, especially when evalu- ating multiple systems. To overcome the latter challenge, we propose a novel application of online learning that, given an ensemble of Ma- chine Translation systems, dynamically con- verges to the best systems, by taking advantage of the human feedback available. Our experi- ments on WMT’19 datasets show that our on- line approach quickly converges to the top-3 ranked systems for the language pairs consid- ered, despite the lack of human feedback for many translations. 1 Introduction In Machine Translation (MT), measuring the qual- ity of a large amount of automatic translations can be a challenge. Automatic metrics like BLEU (Pap- ineni et al., 2002) remain popular due to their fast and free computations. Yet, in the last few years we have seen that, as MT quality improves, auto- matic metrics become less reliable (Ma et al., 2019; Mathur et al., 2020). For example, in the Con- ference on Machine Translation (WMT)’19 News Translation shared task, the winning system accord- ing to human annotators was not even in the top-5 according to BLEU (Barrault et al., 2019). On the other hand, using human assessments can be expen- sive, especially when evaluating multiple systems. In a real world scenario, given an arbitrary number of MT systems, one would need to evaluate them individually to find the best systems for a given lan- guage pair. However, that requires a considerable effort and there may not be enough human anno- tators to evaluate all the systems’ translations. For instance, in the aforementioned WMT’19 shared task, many translations from the competing sys- tems did not receive any human assessment. Given an ensemble of competing, independent MT systems, how can we dynamically find the best ones for a given language pair, while making the most of existing human feedback? To address this question, we present a novel application of online learning to MT: each MT system in the ensemble is assigned to a weight, and the systems’ weights are updated considering human feedback regarding the quality of their translations at each iteration. We use online learning algorithms with theoretical performance guarantees, under the frameworks of prediction with expert advice (Cesa-Bianchi and Lugosi, 2006) …

Content truncated to 500 words