ICL’s Submission to the WMT21 Critical Error Detection Shared Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 928–934 November 10–11, 2021. ©2021 Association for Computational Linguistics 928 ICL’s Submission to the WMT21 Critical Error Detection Shared Task Genze Jiang Zhenhao Li Lucia Specia Language and Multimodal AI (LAMA) Lab, Imperial College London, UK {genze.jiang20, zhenhao.li18, l.specia}@imperial.ac.uk Abstract This paper presents Imperial College London’s submissions to the WMT21 Quality Estima- tion (QE) Shared Task 3: Critical Error De- tection. Our approach builds on cross-lingual pre-trained representations in a sequence clas- sification model. We improve the base classi- fier by (i) adding a weighted sampler to deal with imbalanced data and (ii) introducing fea- ture engineering, where features related to tox- icity, named-entities and sentiment, which are potentially indicative of critical errors, are ex- tracted using existing tools and integrated to the model in different ways. We train models with one type of feature at a time and ensem- ble those models that improve over the base classifier on the development set. Our official submissions achieve very competitive results, ranking second for three out of four language pairs. 1 Introduction Critical Error Detection (CED) is a new task which has been introduced in the WMT21 Quality Estima- tion (QE) Shared Task.1 The purpose of CED is to address a challenging problem in Machine Transla- tion (MT): translations produced by state-of-the-art MT systems can be grammatical and fluent but do not always retain the meaning of the source text. More importantly, incorrect translations can be misleading and even have catastrophic conse- quences such as health, safety, legal, or financial implications. However, these can be hard errors to capture by general QE architectures, which have been shown to be prone towards relying mainly on the translated sentence (Sun et al., 2020). According to the Shared Task definition, a crit- ical translation error is a type of error that occurs when the meaning of the translation deviates from source sentence in a critical way. The task data (Section 2.1) includes five categories of such er- rors: deviation in toxicity (TOX), in named entities 1http://statmt.org/wmt21/quality-estimation-task.html (NAM), in sentiment polarity or negation (SEN), or in numbers (NUM), or introduction of health or safety risks (SAF). The baseline model for this task utilises the XLM-RoBERTa (Conneau et al., 2020) for se- quence classification model, following the Mono- TransQuest architecture proposed by Ranasinghe et al. (2020). Inspired by the fact that these five critical error types refer to specific linguistic phe- nomena, we aim to bring additional information to the models on the presence of such phenomena. The intuition is that sentences containing certain types of linguistic features, such as named entities or dates, are more likely to lead to errors. There- fore, we first process the dataset to extract features reflecting the sentences’ toxicity, sentiment and named entities, using off-the-shelf toolkits or APIs (Section 2.2). We then enhance the baseline archi- tecture with this additional information.2 We experiment with two approaches to take the additional features into account, at token and hid- den state levels. …

Content truncated to 500 words