SIT at MixMT 2022: Fluent Translation Built on Giant Pre-trained Models

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1136 - 1144 December 7-8, 2022 ©2022 Association for Computational Linguistics SIT at MixMT 2022: Fluent Translation Built on Giant Pre-trained Models Abdul Rafae Khan, Hrishikesh Kanade, Girish Amar Budhrani, Preet Jhanglani, and Jia Xu Stevens Institute of Technology {akhan4, hkanade, gbudhran, pjhangl1, jxu70}@stevens.edu Abstract This paper describes the Stevens Institute of Technology’s submission for the WMT 2022 Shared Task: Code-mixed Machine Translation (MixMT). The task consisted of two subtasks, subtask 1 Hindi/English to Hinglish and subtask 2 Hinglish to En- glish translation. Our findings lie in the improvements made through the use of large pre-trained multilingual NMT mod- els and in-domain datasets, as well as back- translation and ensemble techniques. The translation output is automatically eval- uated against the reference translations using ROUGE-L and WER. Our system achieves the 1st position on subtask 2 ac- cording to ROUGE-L, WER, and human evaluation, 1st position on subtask 1 ac- cording to WER and human evaluation, and 3rd position on subtask 1 with respect to ROUGE-L metric. 1 Introduction Code-mixing (or code-switching) is the phe- nomenon when another language like Hindi is interleaved with English words in the same sentence. This code-mixed language is mostly used in social media text and is colloquially referred to as Hinglish. Despite Hindi being the fourth most widely spoken language in the world (Lewis, 2009), research in Hinglish trans- lation has been a relatively unexplored task. A major challenge in creating a transla- tion system for code-mixed text is the lim- ited amount of parallel data (Ranathunga et al., 2021). Typical methods use standard back-translation techniques (Sennrich et al., 2015a) for generating synthetic parallel data for training. Massive multilingual neural ma- chine translation (NMT) models have recently been shown to improve the translation perfor- mances for low-resource and even zero-shot set- tings. We propose using such large multilin- gual NMT models for our code-mixed transla- tion tasks. Previous work has only used smaller multilingual architectures (Gautam et al., 2021). We use pre-trained multilingual models trained in up to 200 language directions. We finetune these models for the Hindi to Hinglish and Hinglish to English tasks. One major chal- lenge when using these massive models is the GPU memory constraint. Another issue is the ratio of English and Hinglish words interleaved for each translation output. We use multiple state-of-the-art GPUs with model paralleliza- tion to overcome the memory issue. For the amount of English in the outputs, we tune the model parameters including learning rate, dropout, and the number of epochs to get the optimal translations. Along with these pre-trained multilingual NMT models, we also use additional in- domain data, back-translation to generate ad- ditional parallel data, and using multi-run ensemble to improve the final performance. All these methods gave us an improvement of +24.4 BLEU for Hindi to Hinglish trans- lation (subtask 1) and +28.1 BLEU points for Hinglish to English translation (subtask 2) compared to using only the organizer provided data and the baseline …

Content truncated to 500 words