Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 625–638 November 10–11, 2021. ©2021 Association for Computational Linguistics 625 Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation Diptesh Kanojia1, Marina Fomicheva2, Tharindu Ranasinghe3, Frédéric Blain4, Constantin Or˘asan5, Lucia Specia6 1,5Centre for Translation Studies, University of Surrey 2University of Sheffield 3,4University of Wolverhampton 6Imperial College London 1,5{d.kanojia,c.orasan}@surrey.ac.uk, 2m.fomicheva@sheffield.ac.uk, 3,4{t.d.ranasinghehettiarachchige,f.blain}@wlv.ac.uk, 6l.specia@imperial.ac.uk Abstract Current Machine Translation (MT) systems achieve very good results on a growing vari- ety of language pairs and datasets. However, they are known to produce fluent translation outputs that can contain important meaning er- rors, thus undermining their reliability in prac- tice. Quality Estimation (QE) is the task of au- tomatically assessing the performance of MT systems at test time. Thus, in order to be use- ful, QE systems should be able to detect such errors. However, this ability is yet to be tested in the current evaluation practices, where QE systems are assessed only in terms of their correlation with human judgements. In this work, we bridge this gap by proposing a gen- eral methodology for adversarial testing of QE for MT. First, we show that despite a high cor- relation with human judgements achieved by the recent SOTA, certain types of meaning er- rors are still problematic for QE to detect. Sec- ond, we show that on average, the ability of a given model to discriminate between meaning- preserving and meaning-altering perturbations is predictive of its overall performance, thus potentially allowing for comparing QE sys- tems without relying on manual quality anno- tation. 1 Introduction Quality Estimation (QE) is the task of predicting the quality of Machine Translation (MT) output in the absence of human reference translation. Recent QE models based on multilingual pre-trained repre- sentations (Ranasinghe et al., 2020) have shown im- pressive results achieving up to 0.9 Pearson correla- tion with human judgements of translation quality at sentence level (Specia et al., 2020). Not unlike other NLP systems, QE systems are typically tested on held-out datasets. On the one hand, such evalua- tion usually requires collecting additional human judgements and thus cannot be easily extrapolated to a different usage scenario, for example, a new language pair. On the other hand, evaluation on a given test set can hide performance issues related to the phenomena that are underrepresented in the data but are critical to the reliable performance of the system. Finally, a single statistic capturing over- all performance does not provide any insights on the strengths and weaknesses of a given approach. As a way to overcome these limitations, we explore adversarial evaluation for QE. Specifically, we in- troduce two types of changes to high-quality MT outputs: meaning-preserving perturbations (MPPs) and meaning-altering perturbations (MAPs). In- tuitively, we expect a strong QE system to assign lower scores to the sentences containing MAPs compared to the sentences with MPPs. Based on this intuition, we devise experiments to systemat- ically test a set of five different QE systems by comparing the scores they produce for sentences containing …

Content truncated to 500 words