Neural Machine Translation for Low-Resourced Indian Languages

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3610–3615 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3610 Neural Machine Translation for Low-Resourced Indian Languages Himanshu Choudhary, Shivansh Rao, Rajesh Rohilla Delhi Technological University (Formerly Delhi college of Engineering) himanshu.dce12@gmail.com, rao.shivansh570@gmail.com, rajesh@dce.ac.in Abstract A large number of significant assets are available online in English, which is frequently translated into native languages to ease the information sharing among local people who are not much familiar with English. However, manual translation is a very tedious, costly, and time-taking process. To this end, machine translation is an effective approach to convert text to a different language without any human involvement. Neural machine translation (NMT) is one of the most proficient translation techniques amongst all existing machine translation systems. In this paper, we have applied NMT on two of the most morphological rich Indian languages, i.e. English-Tamil and English-Malayalam. We proposed a novel NMT model using Multihead self-attention along with pre-trained Byte-Pair-Encoded (BPE) and MultiBPE embeddings to develop an efficient translation system that overcomes the OOV (Out Of Vocabulary) problem for low resourced morphological rich Indian languages which do not have much translation available online. We also collected corpus from different sources, addressed the issues with these publicly available data and refined them for further uses. We used the BLEU score for evaluating our system performance. Experimental results and survey confirmed that our proposed translator (24.34 and 9.78 BLEU score) outperforms Google translator (9.40 and 5.94 BLEU score) respectively. Keywords: Multihead self-attention, Byte-Pair-Encodding, MultiBPE, low-resourced, Morphology, Indian Languages 1. Introduction Many populated countries such as India and China have several languages which change region by region. for example, India has 23 constitutionally recognized official languages (e.g., Hindi, Malayalam, Telugu, Tamil, and Punjabi) and numerous unofficial local languages. Not only big countries, even small countries also rich in language diversity. There are 851 languages spoken in Papua New Guinea, which is one of the smallest populated regions. In India, the population is about three billion, but only about 10% of them can speak English1. Some studies say that out of those 10% English speakers only 2% can talk, write, and examine English well, and rest 8% can merely recognize simple English and talk with a variety of accents. Thinking about a large number of valuable sources is available on the web in English and most people in India can not understand it well, it becomes important to translate such content into neighborhood languages to facilitate people. Sharing pieces of information between human beings is important not only for business purposes but also for sharing their emotions, reviews, and acts. For this, translation plays an essential role in minimizing the communication hole between different peoples. consider- ing the vast amount of text, it is not viable to translate them manually. Hence, it becomes crucial to translate text from one language (say, English) to other languages (say, Tamil, Malayalam) automatically. This technique is also referred to as …

Content truncated to 500 words