A Multilingual Parallel Corpora Collection Effort for Indian Languages

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3743–3751 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3743 A Multilingual Parallel Corpora Collection Effort for Indian Languages Shashank Siripragada*†, Jerin Philip*†, Vinay P. Namboodiri‡, C.V. Jawahar† †International Institute of Information Technology - Hyderabad ‡Indian Institute of Technology - Kanpur {shashank.siripragada@alumni,jerin.philip@research}.iiit.ac.in, vinaypn@iitk.ac.in, jawahar@iiit.ac.in Abstract We present sentence aligned parallel corpora across 10 Indian Languages - Hindi, Telugu, Tamil, Malayalam, Gujarati, Urdu, Bengali, Oriya, Marathi, Punjabi, and English - many of which are categorized as low resource. The corpora are compiled from online sources which have content shared across languages. The corpora presented significantly extends present resources that are either not large enough or are restricted to a specific domain (such as health). We also provide a separate test corpus compiled from an independent online source that can be independently used for validating the performance in 10 Indian languages. Alongside, we report on the methods of constructing such corpora using tools enabled by recent advances in machine translation and cross-lingual retrieval using deep neural network based methods. Keywords: Machine Translation, Parallel Corpus, Indian languages 1. Introduction and Related Work Modern day neural network based approaches for Ma- chine Translation (MT) are data hungry and sentence-level aligned parallel pairs are the currency. Neural MT (NMT) is currently the de-facto approach for training translation systems and shares the same traits (Koehn and Knowles, 2017). Koehn (2005) uses European parliament proceedings available on the web to create evolving parallel corpora. The resource has been a major driving factor in attempts to build MT systems. We provide a similar effort through this work in providing multilingual parallel corpora for Indian languages. Many languages spoken in the Indian subcontinent are categorized as low-resource (Post et al., 2012) consider- ing the amount of parallel corpora available for training with deep neural network based models. There have been a few efforts towards addressing this lacuna. Among this, the IIT-Bombay Hindi English Parallel Corpus (IITB-hi- en) (Kunchukuttan et al., 2017) is the largest English-Hindi corpus available for training. Other significant efforts in constructing parallel corpora for Indian languages include Indian Language Corpora Initiative (ILCI) (Jha, 2010) across 10 Indian languages, restricted to health and tourism domains and Indic Multilingual Parallel Corpus (WAT- ILMPC) (Nakazawa et al., 2017), with possible shared con- tent among multiple languages collected through automated efforts are limited and noisy (Philip et al., 2019). WAT- ILMPC is comprised of user contributed translations of sub- titles, which are inadequate and often code-mixed. A spe- cific effort is that of the Oriya-English Corpus (OdEnCorp) that provides a compilation in Oriya-English (Parida et al., 2020). Lack of resources reflects in the availability of au- tomated translation systems of satisfactory quality in these languages. While some systems are reported (Wang et al., 2017; Philip et al., 2019) to perform well for English-Hindi, *Equal contribution. there are not many that generalize well for other languages. Based on these resources, in order to …

Content truncated to 500 words