Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages

Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages Gowtham Ramesh1∗ Sumanth Doddapaneni1∗ Aravinth Bheemaraj2,5 Mayank Jobanputra3 Raghavan AK4 Ajitesh Sharma2,5 Sujit Sahoo2,5 Harshita Diddee4 Mahalakshmi J4 Divyanshu Kakwani3,4 Navneet Kumar2,5 Aswin Pradeep2,5 Srihari Nagaraj2,5 Kumar Deepak2,5 Vivek Raghavan5 Anoop Kunchukuttan4,6 Pratyush Kumar1,3,4 Mitesh Shantadevi† Khapra1,3,4‡ 1RBCDSAI, India 2Tarento Technologies, India 3IIT Madras, India 4AI4Bharat, India 5EkStep Foundation, India 6Microsoft, India Abstract We present Samanantar, the largest publicly available parallel corpora collection for Indic languages. The collection contains a total of 49.7 million sentence pairs between English and 11 Indic languages (from two language families). Specifically, we compile 12.4 mil- lion sentence pairs from existing, publicly available parallel corpora, and additionally mine 37.4 million sentence pairs from the Web, resulting in a 4× increase. We mine the parallel sentences from the Web by com- bining many corpora, tools, and methods: (a) Web-crawled monolingual corpora, (b) doc- ument OCR for extracting sentences from scanned documents, (c) multilingual repre- sentation models for aligning sentences, and (d) approximate nearest neighbor search for searching in a large collection of sentences. Human evaluation of samples from the newly mined corpora validate the high quality of the parallel sentences across 11 languages. Fur- ther, we extract 83.4 million sentence pairs between all 55 Indic language pairs from the English-centric parallel corpus using English as the pivot language. We trained multilingual NMT models spanning all these languages on Samanantar which outperform existing models and baselines on publicly available benchmarks, such as FLORES, establishing the utility of Samanantar. Our data and mod- els are available publicly at Samanantar and we hope they will help advance research in NMT and multilingual NLP for Indic languages. 1 Introduction The advent of deep-learning (DL) based neural encoder-decoder models has led to significant ∗The first two authors have contributed equally. †Corresponding author: miteshk@cse.iitm.ac.in. ‡Dedicated to the loving memory of my grandmother. progress in machine translation (MT) (Bahdanau et al., 2015; Wu et al., 2016; Sennrich et al., 2016b,a; Vaswani et al., 2017). While this has been favorable for resource-rich languages, there has been limited benefit for resource-poor lan- guages which lack parallel corpora, monolingual corpora, and evaluation benchmarks (Koehn and Knowles, 2017). Multilingual models can im- prove performance on resource-poor languages via transfer learning from resource-rich languages (Firat et al., 2016; Johnson et al., 2017b; Kocmi and Bojar, 2018), more so when the resource-rich and resource-poor languages are related (Nguyen and Chiang, 2017; Dabre et al., 2017). However, it is difficult to achieve this with limited in-language data (Guzm´an et al., 2019), particularly when an entire group of related languages is low-resource, making transfer-learning infeasible. A case in point is that of languages from the Indian subcontinent, a very linguistically diverse region. India has 22 constitutionally listed lan- guages spanning 4 major language families. Other countries in the subcontinent also have their share of widely spoken languages. These languages are closely related both genetically and through con- tact, which led to significant sharing of vocabulary and linguistic features (Emeneau, …

Content truncated to 500 words