NICT Kyoto Submission for the WMT’21 Quality Estimation Task: Multimetric Multilingual Pretraining for Critical Error Detection

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 136–139 November 10–11, 2021. ©2021 Association for Computational Linguistics 136 Miðeind’s WMT 2021 submission Haukur Barri Símonarson, Vésteinn Snæbjarnarson, Pétur Orri Ragnarsson, Haukur Páll Jónsson and Vilhjálmur Þorsteinsson Miðeind ehf., Reykjavík, Iceland {haukur, vesteinn, petur, haukurpj, vt}@mideind.is Abstract We present Miðeind’s submission for the English→Icelandic and Icelandic→English subsets of the 2021 WMT news translation task. Transformer-base models are trained for translation on parallel data to generate back- translations iteratively. A pretrained mBART- 25 model is then adapted for translation using parallel data as well as the last backtransla- tion iteration. This adapted pretrained model is then used to re-generate backtranslations, and the training of the adapted model is continued. 1 Introduction Our work on machine translation for Icelandic has been going on for a couple of years as a part of the government sponsored Icelandic Language Tech- nology Programme (Nikulásdóttir et al., 2020). By building on state-of-the-art solutions we have de- veloped an open and effective translation system between Icelandic and English. To achieve this, we collect parallel Icelandic and English texts which are filtered for good quality alignments. We also collect monolingual text for backtranslations. We follow tried and tested meth- ods in neural machine translation using iterative backtranslation (Edunov et al., 2018) and adapt the multilingual denoising autoencoder model mBART- 25 (Liu et al., 2020) for translation. 2 Datasets We used several parallel and monolingual datasets, both publicly available and created in-house. 2.1 Parallel data The parallel data used are ParIce (Steingrímsson et al., 2020) and the JW300 corpus (Agi´c and Vuli´c, 2019). In addition we used a parallel student theses and dissertation abstracts corpus, IPAC, generated in-house and sourced from the Skemman reposi- tory1 as described in (Símonarson and Snæbjarnar- son, 2021). A breakdown of the data is shown in Table 1. Corpus #Sentences The Bible 33k EEA regulatory texts 1,700k EMA 404k ESO 12.6k OpenSubtitles 1,300k Tatoeba 10k Jehova’s Witnesses (JW300) 527k Other∗ 93k IPAC 64k Table 1: Parallel corpora used. #Sentences are the num- ber of sentence pairs. Other∗ denotes software local- izations, Project Gutenberg literature and the Icelandic sagas. Following (Pinnis, 2018) we apply simple heuris- tic filters to the parallel data, mainly for capturing OCR and PDF errors, and correcting or removing character encoding errors after deduplication. Fil- ters include but are not limited to: empty sequence removal, length cut-offs, character whitelists, mis- match in case and symbols between languages, edit- distances between source and target, normalizing of punctuation, and ad-hoc regular expressions for Icelandic specific OCR/PDF errors. For a more in-depth description see (Jónsson et al., 2020). Other potential parallel datasets are ParaCrawl (Bañón et al., 2020) and CCMatrix (Schwenk et al., 2021). Manual review of a couple of hundred ran- domly chosen lines from ParaCrawl revealed that the data quality is quite low for Icelandic, many lines are machine translated or badly aligned. We therefore did not include ParaCrawl. CCMatrix did not exist when the project started and we have not …

Content truncated to 500 words