Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799–2808 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2799 Putting Evaluation in Context: Contextual Embeddings improve Machine Translation Evaluation Nitika Mathur Timothy Baldwin Trevor Cohn School of Computing and Information Systems The University of Melbourne Victoria 3010, Australia nmathur@student.unimelb.edu.au {tbaldwin,tcohn}@unimelb.edu.au Abstract Accurate, automatic evaluation of machine translation is critical for system tuning, and evaluating progress in the field. We proposed a simple unsupervised metric, and additional supervised metrics which rely on contextual word embeddings to encode the translation and reference sentences. We find that these models rival or surpass all existing metrics in the WMT 2017 sentence-level and system- level tracks, and our trained model has a substantially higher correlation with human judgements than all existing metrics on the WMT 2017 to-English sentence level dataset. 1 Introduction Evaluation metrics are a fundamental compo- nent of machine translation (MT) and other lan- guage generation tasks. The problem of assess- ing whether a translation is both adequate and coherent is a challenging text analysis problem, which is still unsolved, despite many years of ef- fort by the research community. Shallow surface- level metrics, such as BLEU and TER (Papineni et al., 2002; Snover et al., 2006), still predominate in practice, due in part to their reasonable corre- lation to human judgements, and their being pa- rameter free, making them easily portable to new languages. In contrast, trained metrics (Song and Cohn, 2011; Stanojevic and Sima’an, 2014; Ma et al., 2017; Shimanaka et al., 2018), which are learned to match human evaluation data, have been shown to result in a large boost in performance. This paper aims to improve over existing MT evaluation methods, through developing a series of new metrics based on contextual word embed- dings (Peters et al., 2018; Devlin et al., 2019), a technique which captures rich and portable repre- sentations of words in context, which have been shown to provide important signal to many other NLP tasks (Rajpurkar et al., 2018). We pro- pose a simple untrained model that uses off-the- shelf contextual embeddings to compute approx- imate recall, when comparing a reference to an automatic translation, as well as trained mod- els, including: a recurrent model over reference and translation sequences, incorporating atten- tion; and the adaptation of an NLI method (Chen et al., 2017) to MT evaluation. These approaches, though simple in formulation, are highly effec- tive, and rival or surpass the best approaches from WMT 2017. Moreover, we show further improve- ments in performance when our trained models are learned using noisy crowd-sourced data, i.e., hav- ing single annotations for more instances is bet- ter than collecting and aggregating multiple anno- tations for single instances. The net result is an approach that is more data efficient than existing methods, while producing substantially better hu- man correlations.1 2 Related work MT metrics attempt to automatically predict the quality of a translation by comparing it to a refer- ence …

Content truncated to 500 words