A3-108 Machine Translation System for Similar Language Translation Shared Task 2021

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 304–306 November 10–11, 2021. ©2021 Association for Computational Linguistics 304 A3-108 Machine Translation System for Similar Language Translation Shared Task 2021 Saumitra Yadav, Manish Shrivastava Machine Translation - Natural Language Processing Lab Language Technologies Research Centre Kohli Center on Intelligent Systems International Institute of Information Technology - Hyderabad saumitra.yadav@research.iiit.ac.in m.shrivastava@iiit.ac.in Abstract In this paper, we describe our submissions for the Similar Language Translation Shared Task 2021. We built 3 systems in each direction for the Tamil ⇐⇒ Telugu language pair. This paper outlines experiments with various tok- enization schemes to train statistical models. We also report the configuration of the submit- ted systems and results produced by them. 1 Introduction Machine translation is a process of translating text from a source to a target language. There are mul- tiple ways of building such a system - Rule-based, Data-driven, Hybrid etc. In this shared task, we use data-driven method to create machine translation system for Tamil ⇐⇒ Telugu. Due to low-resource setting of this language pair in the shared task, we use Statistical Machine translation method (Koehn et al., 2003),(Koehn and Knowles, 2017) to build systems. Tamil Telugu language pair comes under the bracket of similar languages. Similar languages show similarity in their lexical and syntactical prop- erties (Kunchukuttan et al., 2014a). This may be due to them being in close proximity of each other for long time. This can also be due to common ancestry. In the current digital context, transla- tion between similar languages is of importance. But there can be scarcity of good quality parallel text. In the current shared task, we have a language pair which is morphologically rich and with ≃39K parallel sentences. So, following Kunchukuttan and Bhattacharyya (2017) and Kunchukuttan et al. (2014b) we use sentencepiece1 (Kudo and Richard- son, 2018) and morfessor2 (Virpioja et al., 2013) to segment tokens in the dataset into subwords. And due to the size of parallel text (≃39K parallel 1https://github.com/google/ sentencepiece 2https://github.com/aalto-speech/ morfessor text) coming under purview of low resource, we make use of Moses3(Koehn et al., 2007) to create statistical machine translation models(Koehn and Knowles, 2017). For this shared task we developed 3 translation systems (1 Primary and 2 Contrastive) in each direc- tion Tamil ⇐⇒ Telugu. For each output we post- processed and detokenized translation output de- pending on the tokenization scheme for target lan- guage. To choose a primary and 2 contrastive sys- tems, we compared BLEU (Papineni et al., 2002) scores on output of development dataset for each system using sacrebleu4 (Post, 2018). The Follow- ing sections give more details about the systems developed. 2 SMT systems using different schemes We used various tokenization schemes to build translation systems. Evaluated these systems on the development dataset. After post-processing, deto- kenizing and scoring each translation output, we submit output systems as primary and contrastive submissions accordingly. 2.1 Data and preprocessing We used parallel data provided by the organizers to train all the models. IndicNLP5 (Kunchukut- tan, …

Content truncated to 500 words