Samsung R&D Institute Poland Participation in WMT 2022

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 251 - 259 December 7-8, 2022 ©2022 Association for Computational Linguistics Samsung R&D Institute participation in WMT 2022 General MT Task Adam Dobrowolski1, Mateusz Klimaszewski∗2, Adam My´sliwy1, Marcin Szyma´nski1, Jakub Kowalski1, Kornelia Szypuła1, Paweł Przewłocki1, Paweł Przybysz1 1Samsung R&D Institute, Warsaw, Poland 2Warsaw University of Technology, Warsaw, Poland {a.dobrowols2, a.mysliwy, m.szymanski, j.kowalski5, k.szypula, p.przybysz}@samsung.com mateusz.klimaszewski.dokt@pw.edu.pl, p.przewlocki@partner.samsung.com Abstract This paper presents the system description of Samsung R&D Institute Poland participation in WMT 2022 for General MT solution for medium and low resource languages: Russian and Croatian. Our approach combines iterative noised/tagged back-translation and iterative dis- tillation. We investigated different monolingual resources and compared their influence on fi- nal translations. We used available BERT-like models for text classification and for extract- ing domains of texts. Then we prepared an ensemble of NMT models adapted to multiple domains. Finally we attempted to predict en- semble weight vectors from the BERT-based domain classifications for individual sentences. Our final trained models reached quality com- parable to best online translators using only limited constrained resources during training. 1 Introduction Samsung R&D Institute Poland (SRPOL) partic- ipated in the WMT 2022 General MT task for three translation directions: EN→RU, RU→EN and EN→HR. All our systems were built using only constrained datasets. In contrast to previous years, where the task focused on news translation, this year’s task was domain-independent. However, MT models benefit a lot from domain adaptation. Therefore, we decided to prepare an ensemble of NMT models adapted to multiple domains to bene- fit from domain adaptation and improve generaliza- tion. We prepared a news profiled model but also a general-purpose one. Additionally, we worked on medical and legal domains; however, there was very limited in-domain data in the constraint path for this domains and we had to extract pseudo in- domain data from monolingual corpora. Our system was implemented using Marian framework. The core of the submitted solution is iterative back-translation and iterative distillation ∗Work done while at Samsung R&D Institute Poland. combined with finetuning and ensembling. Besides, we used BERT models for data filtering to prepare corpora for training domain-adapted models. Fi- nally, we created dynamic ensemble weighting to choose the best combination of single models in the final translations. All techniques combined allowed us to improve baseline models by 3-6 BLEU (Pap- ineni et al., 2002) and reach the quality comparable with online translators (measured by BLEU). 2 System overview 2.1 MT model Our models were trained with the Marian NMT (Junczys-Dowmunt et al., 2018) toolkit. We used Marian for training, back-translation, noise genera- tion, language models and data filtering. The training was performed on a transformer-big model (embedding dimension of 1024 and a feed- forward layer dimension of 4096) (Vaswani et al., 2017). We experimented with different sizes of models and different configurations of encoder- decoder layers, but we achieved no significant im- provement over the default transformer-big con- figuration. Most models had a setup of either 7-5 or 8-4 encoder-decoder layers. Best …

Content truncated to 500 words