TUDa at WMT21: Sentence-Level Direct Assessment with Adapters

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 911–919 November 10–11, 2021. ©2021 Association for Computational Linguistics 911 TUDa at WMT21: Sentence-Level Direct Assessment with Adapters Gregor Geigle, Jonas Stadtmüller, Wei Zhao, Jonas Pfeiffer, Steffen Eger Technische Universität Darmstadt {gregortheodor.geigle,jonas.stadtmueller}@stud.tu-darmstadt.de {zhao,eger}@aiphes.tu-darmstadt.de pfeiffer@ukp.informatik.tu-darmstadt.de Abstract This paper presents our submissions to the WMT2021 Shared Task on Quality Estima- tion, Task 1 Sentence-Level Direct Assess- ment. While top-performing approaches uti- lize massively multilingual Transformer-based language models which have been pre-trained on all target languages of the task, the re- sulting insights are limited, as it is unclear how well the approach performs on languages unseen during pre-training; more problemati- cally, these approaches do not provide any so- lutions for extending the model to new lan- guages or unseen scripts—arguably one of the objectives of this shared task. In this work, we thus focus on utilizing massively multilin- gual language models which only partly cover the target languages during their pre-training phase. We extend the model to new languages and unseen scripts using recent adapter-based methods and achieve on par performance or even surpass models pre-trained on the respec- tive languages. 1 Introduction In Machine Translation (MT), the Quality Estima- tion (QE) task attempts to characterize the quality of a translation, without the availability of a (gold- label) reference translation. The introduction of a QE system would consequently allow for the au- tomatic analysis of machine-translated sentences without costly human reference translation, with numerous applications, such as: the selection of candidate translations, the estimation of human editing effort, or the detection of low-quality or misleading translations (Kepler et al., 2019). How- ever, in order to acquire training data, professional human translators are required to score the trans- lation quality of many examples, making labeled data difficult to obtain, especially for low-resource languages. This highlights the importance of cross- lingual zero-shot transfer of QE systems, one of the objectives of the WMT21 shared task (Specia et al., 2021), which introduces zero-shot evaluation sets of four new language pairs. Previous approaches have predominantly fo- cused on languages for which training data is available, such as the QE task at WMT20. The best results were obtained by fine-tuning massively multilingual Transformer-based language models (Vaswani et al., 2017) such as multilingual BERT (mBERT) (Devlin et al., 2019) or XLM-R (Con- neau et al., 2020) (Specia et al., 2020; Ranasinghe et al., 2020b; Sun et al., 2020a; Nakamachi et al., 2020, inter alia), on the target QE tasks. These supervised methods considerably outperform un- supervised methods (Zhao et al., 2020; Fomicheva et al., 2020c; Sun et al., 2020a; Zhao et al., 2021; Song et al., 2021) even in zero-shot settings (Sun et al., 2020a). However, analyzing the applicabil- ity of fine-tuning multilingual models on the target language pairs that are covered during pre-training considerably limits the generated insights. They are only applicable to the ∼100 languages covered during pre-training, excluding the remaining ma- jority of languages as the “curse-of-multilinguality” (Conneau et al., 2020) prohibits the over 7000 …

Content truncated to 500 words