Hindi-Marathi Cross Lingual Model

Proceedings of the 5th Conference on Machine Translation (WMT), pages 396–401 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 396 Hindi-Marathi Cross Lingual Model Sahinur Rahman Laskar, Abdullah Faiz Ur Rahman Khilji, Partha Pakray, Sivaji Bandyopadhyay Department of Computer Science and Engineering National Institute of Technology Silchar Assam, India {sahinur_rs, abdullah_ug, partha}@cse.nits.ac.in, sivaji.cse.ju@gmail.com Abstract Machine translation (MT) is a vital tool for aiding communication between linguistically separate groups of people. The neural ma- chine translation (NMT) based approaches have gained widespread acceptance because of its outstanding performance. We have partic- ipated in WMT20 shared task of similar lan- guage translation on Hindi-Marathi pair. The main challenge of this task is by utilization of monolingual data and similarity features of similar language pair to overcome the limita- tion of available parallel data. In this work, we have implemented NMT based model that si- multaneously learns bilingual embedding from both the source and target language pairs. Our model has achieved Hindi to Marathi bilingual evaluation understudy (BLEU) score of 11.59, rank-based intuitive bilingual evaluation score (RIBES) score of 57.76 and translation edit rate (TER) score of 79.07 and Marathi to Hindi BLEU score of 15.44, RIBES score of 61.13 and TER score of 75.96. 1 Introduction MT is a well-known task of natural language pro- cessing (NLP) wherein automatic translation is per- formed between different languages. Broadly, MT is categorized into rule-based and corpus-based, where rule-based is based on a pre-defined rules on the concerned languages and corpus-based finds a generalized approach after being trained on a large corpus. MT switches from rule-based approach to the corpus-based which blots out the need for lin- guistic expertise. In the corpus-based approach, example-based machine translation (EBMT), sta- tistical machine translation (SMT) and NMT tech- niques are available. The disadvantage of EBMT is that even though the corpus is large, all exam- ples are not covered. To mitigate the issues of the contemporary approach SMT is introduced Brown et al. (1990); Koehn (2010). The SMT based system makes an assumption based on probabil- ity scores of the translated text. And hence, the ranking is done. SMT also faces many issues like system complexity, long term dependency prob- lem, context-analyzing inability, word-alignment and the rare word problem. The inefficiency of SMT leads to the development of the NMT De- vlin et al. (2014). But like SMT, the NMT based model also suffers the requirement of sufficient training parallel corpus, which is a challenge in the case of low resource languages. For this reason, there is a demand for direct translation among sim- ilar language pairs by utilizing similarity features and monolingual data, so that less availability of the parallel data does not pose a challenge. How- ever, the NMT technique achieves state-of-the-art approach in MT because of its transformer model Vaswani et al. (2017). For low resource language pair translation, NMT models have been improved with monolingual corpus Sennrich et al. (2016b); Burlot and Yvon (2018); Wu et al. (2019). In this work, we have …

Content truncated to 500 words