The University of Maryland, College Park Submission to Large-Scale Multilingual Shared Task at WMT 2021

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 383–386 November 10–11, 2021. ©2021 Association for Computational Linguistics 383 The University of Maryland, College Park Submission to Large-Scale Multilingual Shared Task at WMT 2021 Saptarashmi Bandyopadhyay∗ Tasnim Kabir∗ Zizhen Lian∗ Marine Carpuat Department of Computer Science University of Maryland, College Park {saptab1,tkabir1,zizhlian,marine}@umd.edu Abstract This paper describes the system submitted to Large-Scale Multilingual Shared Task (Small Task #2) at WMT 2021. It is based on the mas- sively multilingual open-source model FLO- RES101_MM100 model, with selective fine- tuning. Our best-performing system reported a 15.72 average BLEU score for the task. 1 Introduction Massively multilingual models such as Facebook’s M2M-100 (Fan et al., 2020) model provide an at- tractive approach to scaling Machine Translation to many language pairs by sharing encoder-decoder parameters across languages. By not centering En- glish in its training process, M2M-100 improves translation quality substantially (by over 10 BLEU points) compared to the best single systems of WMT before 2020 on the “large-scale Many-to- Many dataset for 100 languages” (Fan et al., 2020). However, translation quality for low-resource lan- guages still leaves much room for improvement. We address the Large-Scale Multilingual Ma- chine Translation Shared Task (Small Track #2) at WMT 2021, by fine-tuning the FLO- RES101_MM100 model for the languages in the Shared task. We consider different fine-tuning con- figurations, with a goal to minimize the compu- tational and data resources required. First, we consider the impact of finetuning on datasets of different sizes, and surprisingly show that finetun- ing with the smaller dataset gives better perfor- mance for some language pairs. Second, we con- sider selectively dropping layers during fine-tuning to reduce the computational cost of working with a Transformer model with millions of parameters. We adopt a structure dropout technique, LayerDrop, which has been shown to have a regularization ef- fect and to effectively reduce model size for infer- ence (Fan et al., 2019), as well as to reduce training *These authors contributed equally to this work time while preserving decoding quality (Zhang and He, 2020). We have used LayerDrop so that our model can run on large datasets for low resource language pairs. Our best performing system is fine-tuned on the large MultiCCAligned training data and yields a sentence-piece BLEU score (the official Shared task metric) of 15.72 on the Shared task test set. However, a model fine-tuned on smaller amounts of data (bible-uedin) approaches that result, with a BLEU score of 15.10. This paper describes the submitted models, as well as experiments with Lay- erDrop configuration, which show that dropping the top layers does not help BLEU. 2 Shared Task Data Training Our training data is provided by the Shared task organizers and is drawn from the pub- licly available open-source multilingual parallel corpus (OPUS) data repository for the languages of the Shared task (Tiedemann, 2012). It consists of the MultiCCAligned large dataset which sup- ports 112 languages (El-Kishky et al., 2020) with English as the pivot language. The bible-uedin dataset (Christodouloupoulos and …

Content truncated to 500 words