IST-Unbabel Participation in the WMT20 Quality Estimation Shared Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1029–1036 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1029 IST-Unbabel Participation in the WMT20 Quality Estimation Shared Task Jo˜ao Moura Instituto Superior T´ecnico, Lisbon joaopcmoura@tecnico.ulisboa.pt Miguel Vera Unbabel, Lisbon miguel.vera@unbabel.com Daan van Stigt Unbabel, Lisbon daan.stigt@unbabel.com Fabio Kepler Unbabel, Lisbon kepler@unbabel.com Andr´e F. T. Martins Instituto de Telecomunicac¸˜oes Instituto Superior T´ecnico Unbabel, Lisbon andre.t.martins@tecnico.ulisboa.pt Abstract We present the joint contribution of IST and Unbabel to the WMT 2020 Shared Task on Quality Estimation. Our team participated on all tracks (Direct Assessment, Post-Editing Ef- fort, Document-Level), encompassing a total of 14 submissions. Our submitted systems were developed by extending the OpenKiwi framework to a transformer-based predictor- estimator architecture, and to cope with glass- box, uncertainty-based features coming from neural machine translation systems. 1 Introduction Quality estimation (QE) is the task of evaluating a translation system’s quality without access to refer- ence translations (Blatz et al., 2004; Specia et al., 2018). This paper describes the joint contribution for Instituto Superior T´ecnico (IST) and Unbabel to the WMT20 Quality Estimation shared task, where systems were submitted to all three tasks: 1) sentence-level direct assessment; 2) word and sentence-level post-editing effort; and 3) document- level annotation and scoring. Unbabel’s participation in previous editions of the shared task (2016, 2017, 2019) used ensemble of strong individual systems, with varying architec- tures and hyper-parameters. While this strategy led to very strong results, large system ensembles are not a very practical solution, complicating model deployment and requiring expensive computation and memory usage. This year, in contrast, our fo- cus was on simplicity: only single model systems were submitted and, in a few cases, an additional simple ensemble of the same model. Transfer learn- ing on top of pretrained multilingual models was also used for avoiding manual pretraining for each language pair. Last year’s winning submission (Kepler et al., 2019a) combined strong individual systems built on top of the OpenKiwi framework (Kepler et al., 2019b) and pretrained Transformer models. We consolidated those changes with support for newly released pretrained models and packages and pub- lished a new version 2.0 of the OpenKiwi frame- work.1 We trained and submitted single model systems in OpenKiwi for all tasks, beating all base- lines by a large margin. Additionaly, we also used OpenKiwi with small adaptations to handle specific sources of information in Tasks 1 and 3. Task 1, in particular, was introduced this year with Direct Assessment scores as targets. Further, it introduced the novelty of providing the trained NMT models that were used for producing the translations. Previously, only black-box QE was considered in the WMT Shared Task, as it is one of the main uses cases. With the availability of the NMT models, new glass-box approaches can be explored. Our best submitted systems drew inspi- ration from (Fomicheva et al., 2020) to leverage this information, improving in performance and robustness over a black-box approach. Our main contributions are: • We release the second version of OpenKiwi along …

Content truncated to 500 words