NICT at MixMT 2022: Synthetic Code-Mixed Pre-training and Multi-way Fine-tuning for Hinglish–English Translation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1122 - 1125 December 7-8, 2022 ©2022 Association for Computational Linguistics NICT at MixMT 2022: Synthetic Code-Mixed Pre-training and Multi-way Fine-tuning for Hinglish–English Translation Raj Dabre National Institute of Information and Communications Technology (NICT) Kyoto, Japan raj.dabre@nict.go.jp Abstract In this paper, we describe our submission to the Code-mixed Machine Translation (MixMT) shared task. In MixMT, the objective is to trans- late Hinglish to English and vice versa. For our submissions, we focused on code-mixed pre-training and multi-way fine-tuning. Our submissions achieved rank 4 in terms of auto- matic evaluation score. For Hinglish to English translation, our submission achieved rank 4 as well. 1 Introduction Code-mixed translation is the task of translation involving code-mixed languages. A code-mixed language is one which combines words as well as grammar of two or more languages. Code-mixed translation is difficult because of the lack of train- ing data for the same despite its ubiquitous usage. One widely used code-mixed language is Hinglish which combines Hindi and English. Hinglish sen- tences are typically constructed either by replacing some Hindi words or phrases with English ones in a Hindi sentence or vice versa. Sometimes, a sen- tence starts off in one language but ends in another. There are also complex cases where the grammat- ical structures of both languages are melded into one. Hinglish is typically written in Roman let- ters, although there are cases when it is written in Devanagari. In this paper we describe our submissions to the MixMT task which involves Hinglish to English and English/Hindi to Hinglish translation. The main challenge of this task is that the parallel cor- pus available for training models is rather scarce. The total amount of clean, non-synthetic data avail- able for MixMT is around 18,000 examples for both directions. Therefore, we have no choice but to rely on external sources of data, and use them to pre-train models. In our case, we leverage a large amount of Hindi–English parallel data and synthe- size pseudo Hinglish data. To do this, perform word alignment on the Hindi–English data and then replace random English phrases with aligned Hindi phrases. We then use the synthetic Hinglish– English parallel data for pre-training. The pre- trained model is then fine-tuned to train a joint bidi- rectional Hinglish–English translation model. Ac- cording to the automatic evaluation metrics, we ob- tain 4th rank and on human evaluation of Hinglish to English translation, we also obtain 4th rank. Un- fortunately, for translation into Hinglish our sys- tem ends up copying the English inputs as outputs. Although automatic evaluation scores for this are reasonably high, their human evaluation scores are lowest since the sentences are not Hinglish at all. 2 Related Work Work on code-mixed machine translation is rela- tively new, especially for Hinglish. Two impor- tant works in this regard are HinGE (Srivastava and Singh, 2021) which proposes a dataset for En- glish/Hindi to Hinglish translation and PHINC (Sri- vastava and Singh, 2020) which proposes a …

Content truncated to 500 words