IST-Unbabel 2021 Submission for the Quality Estimation Shared Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 961–972 November 10–11, 2021. ©2021 Association for Computational Linguistics 961 IST-Unbabel 2021 Submission for the Quality Estimation Shared Task Chrysoula Zerva1,2,∗ Daan van Stigt3,∗ Ricardo Rei2,3,4,∗ Ana C. Farinha3 Pedro G. Ramos3 José G. C. de Souza3 Taisiya Glushkova1,2 Miguel Vera3 Fábio Kepler3 André F. T. Martins1,2,3 1Instituto Superior Técnico 2Instituto de Telecomunicações 3Unbabel 4INESC-ID 2,3,4Lisbon, Portugal {chrysoula.zerva, ricardo.rei, taisiya.glushkova, andre.t.martins}@tecnico.ulisboa.pt {daan.stigt, catarina.farinha, pedro.ramos, jose.souza, miguel.vera, fabio.kepler}@unbabel.com Abstract We present the joint contribution of IST and Unbabel to the WMT 2021 Shared Task on Quality Estimation. Our team participated on two tasks: Direct Assessment and Post- Editing Effort, encompassing a total of 35 submissions. For all submissions, our ef- forts focused on training multilingual models on top of OpenKiwi predictor-estimator ar- chitecture, using pre-trained multilingual en- coders combined with adapters. We further experiment with and uncertainty-related objec- tives and features as well as training on out-of- domain direct assessment data. 1 Introduction Quality estimation (QE) is the task of evaluating a translation system’s quality without access to ref- erence translations (Blatz et al., 2004; Specia et al., 2018). This paper describes the joint contribution of Instituto Superior Técnico (IST) and Unbabel to the WMT21 Quality Estimation shared task (Spe- cia et al., 2021), where systems were submitted to two tasks: 1) sentence-level direct assessment; 2) word- and sentence-level post-editing effort. This year’s submission combines several ideas built on top of the OpenKiwi framework. Moti- vated by the mixture of blind and seen language pairs in the test sets, we experimented with ex- tensions that would allow us to train multilingual models that maintain good generalization ability and are robust to the presence of epistemic and aleatoric uncertainty. For both tasks we trained and submitted an en- semble of multilingual models. All submitted mod- els follow the predictor-estimator architecture (Kim and Lee, 2016; Kim et al., 2017) and use pre- trained models for feature extraction. Also, we fine-tune all models on the provided QE data us- ing stacked adapter layers (Pfeiffer et al., 2020). ∗ The first three authors have equal contribution. We show that we can thus achieve comparable per- formance across language pairs while minimising the number of trainable parameters (see Table 1). Furthermore, we experimented with different types of uncertainty-related information to leverage it’s benefits, improving performance and robustness of the submitted systems (see §3.1.1). All related code extensions will be publicly available. Our main contributions are: • We build on our OpenKiwi architecture by exploring adapter layers (Houlsby et al., 2019; Pfeiffer et al., 2020) for quality estimation as these demonstrated to be less amenable to overfitting while presenting the same or superior quality performance than fine-tuning the whole base pre-trained model for different NLP tasks (He et al., 2021). • We incorporate different types of uncertainty into our architectures. We make use of the glass-box features (Fomicheva et al., 2020) extracted from the NMT models, the aleatoric (data) uncertainty derived from the human annotations and the …

Content truncated to 500 words