Translation Error Detection as Rationale Extraction

Findings of the Association for Computational Linguistics: ACL 2022, pages 4148 - 4159 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Translation Error Detection as Rationale Extraction Marina Fomicheva University of Sheffield m.fomicheva@sheffield.ac.uk Lucia Specia Imperial College London l.specia@imperial.ac.uk Nikolaos Aletras University of Sheffield n.aletras@sheffield.ac.uk Abstract Recent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results in pre- dicting the overall quality of translated sen- tences. However, detecting specifically which translated words are incorrect is a more chal- lenging task, especially when dealing with lim- ited amounts of training data. We hypothesize that, not unlike humans, successful QE mod- els rely on translation errors to predict overall sentence quality. By exploring a set of fea- ture attribution methods that assign relevance scores to the inputs to explain model predic- tions, we study the behaviour of state-of-the- art sentence-level QE models and show that explanations (i.e. rationales) extracted from these models can indeed be used to detect translation errors. We therefore (i) introduce a novel semi-supervised method for word-level QE; and (ii) propose to use the QE task as a new benchmark for evaluating the plausibility of feature attribution, i.e. how interpretable model explanations are to humans. 1 Introduction Quality Estimation (QE) is the task of predict- ing Machine Translation (MT) quality at inference time, when no gold standard human translation is available (Blatz et al., 2004; Specia et al., 2009). QE can be framed as a word-level or a sentence- level task. Both tasks have numerous practical applications, such as deciding whether a given MT output can be published without editing, highlight- ing potential critical errors. Current QE approaches fine-tune powerful representations from pre-trained multilingual encoders such as BERT (Devlin et al., 2018) or XLM-R (Conneau et al., 2019). In the recent Shared Task on QE at WMT2020 (Specia et al., 2020) these approaches have achieved very high performance at predicting sentence-level trans- lation quality (up to 0.9 Pearson correlation with human judgements for some language pairs). How- ever, as evidenced by these results, the accuracy of word-level prediction still leaves room for improve- ment. This is partly due to the limited amount of training data. Word-level error annotation is espe- cially time-consuming and expensive, as it requires work from bilingual experts. In this work we intro- duce a new semi-supervised approach to word-level QE that removes the need of training data at word level. To achieve this, we propose addressing QE as a rationale extraction task (Lei et al., 2016). Explainability is a broad area aimed at explain- ing predictions of machine learning models (Lip- ton, 2016). Rationale extraction methods achieve this by selecting a portion of the input that justifies model output for a given data point. In transla- tion, human perception of quality is guided by the presence of translation errors (Freitag et al., 2021). We hypothesize that sentence-level QE models also rely on translation errors to make predictions. If that is the case, explanations for sentence-level predictions can be used to …

Content truncated to 500 words