Multi-Hypothesis Machine Translation Evaluation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1218–1232 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1218 Multi-Hypothesis Machine Translation Evaluation Marina Fomincheva1 Lucia Specia1,2 Francisco Guzm´an3 1Department of Computer Science, University of Sheffield, UK 2Department of Computing, Imperial College London, UK 3Facebook AI, Menlo Park, CA, USA m.fomicheva@sheffield.ac.uk l.specia@imperial.ac.uk fguzman@fb.com Abstract Reliably evaluating Machine Translation (MT) through automated metrics is a long-standing problem. One of the main challenges is the fact that multiple outputs can be equally valid. Attempts to minimise this issue include met- rics that relax the matching of MT output and reference strings, and the use of multiple ref- erences. The latter has been shown to sig- nificantly improve the performance of evalu- ation metrics. However, collecting multiple references is expensive and in practice a sin- gle reference is generally used. In this paper, we propose an alternative approach: instead of modelling linguistic variation in human ref- erence we exploit the MT model uncertainty to generate multiple diverse translations and use these: (i) as surrogates to reference transla- tions; (ii) to obtain a quantification of transla- tion variability to either complement existing metric scores or (iii) replace references alto- gether. We show that for a number of popu- lar evaluation metrics our variability estimates lead to substantial improvements in correlation with human judgements of quality by up 15%. 1 Introduction Translation is an open-ended task with multiple valid solutions. There are often multiple equiva- lent translations for the same source sentence. This is due to inherent differences between languages and various sources of ambiguity, which is often impossible to solve without access to additional context. Furthermore, the source might suffer sub- stantial changes in translation due to translator’s need to adapt it to the target audience. With rare exceptions, translations are not literal, they can differ from the source text at any linguistic level – lexical, syntactic, semantic or even discourse – and still be considered correct. The ability to pro- duce non-literal, more natural translations is one of the goals in the field of Machine Translation (MT). Neural MT (NMT) approaches have certainly made significant progress in this direction. However, the diversity of possible outcomes makes it harder to evaluate MT models. Evaluation metrics (or humans in the case of monolingual man- ual evaluation) are given a single reference trans- lation against which to compare the MT output. Fomicheva and Specia (2016) found differences of up to 1 point on a 1-5 point quality scale (i.e. 20%) between groups of annotators who use differ- ent references for manual evaluation. In automatic evaluation, which computes a similarity score be- tween MT output and human reference, they found differences of up to 6 BLEU points depending on the reference used, showing that metrics strongly penalise perfectly correct translations that happen to be different from the reference provided. Dreyer and Marcu (2012) showed that if multi- ple human translations are used, any automatic MT evaluation metric achieves a substantially higher …

Content truncated to 500 words