The IIT Bombay English-Hindi Parallel Corpus

The IIT Bombay English-Hindi Parallel Corpus Anoop Kunchukuttan†, Pratik Mehta1‡, Pushpak Bhattacharyya† †Center for Indian Language Technology, Department of Computer Science and Technology, Indian Institute of Technology Bombay. ‡ College of Information and Computer Sciences, University of Massachusetts Amherst. {anoopk,pb}@cse.iitb.ac.in, psmehta@cs.umass.edu Abstract We present the IIT Bombay English-Hindi Parallel Corpus. The corpus is a compilation of parallel corpora previously available in the public domain as well as new parallel corpora we collected. The corpus contains 1.49 million parallel segments, of which 694k segments were not previously available in the public domain. The corpus has been pre-processed for machine translation, and we report baseline phrase-based SMT and NMT translation results on this corpus. This corpus has been used in two editions of shared tasks at the Workshop on Asian Language Translation (2016 and 2017). The corpus is freely available for non-commercial research. To the best of our knowledge, this is the largest publicly available English-Hindi parallel corpus. Keywords: machine translation, parallel corpus, Indian languages 1. Introduction Hindi is one of the major languages of the world, spo- ken primarily in the Indian subcontinent, and is a recog- nised regional language in Mauritius, Trinidad and Tobago, Guyana, and Suriname. In addition, it serves as a major lingua franca in India. According to the 2001 Census of India, Hindi has 422 million native speakers and more than 500 million total speakers (Wikipedia, 2017). It is also an official language of the Union Government of India as well as major Indian states like Uttar Pradesh, Bihar, Rajasthan, etc. and is used for conducting business and administrative tasks. Many languages and dialects in the Gangetic plains are closely related to Hindi e.g. Bhojpuri, Awadhi, Maithili, etc. Hindi is the fourth-most spoken language in the world, and third-most spoken language along with Urdu (both are registers of the Hindustani language). In contrast, English is spoken by just around 125 million people in India, of which a very small fraction are native speakers. Hence, there is a large requirement for digital communi- cation in Hindi and interfacing with the rest of the word via English. Hence, there is immense potential for English- Hindi machine translation. However, the parallel corpora available in the public domain is quite limited. This work is an effort to consolidate all publicly available parallel cor- pora for English-Hindi as well as significantly add to the available parallel corpus through corpora collected in the course of this work. 2. Dataset The parallel corpus has been compiled from a variety of existing sources (primarily OPUS (Tiedemann, 2012), Hin- dEn (Bojar et al., 2014b) and TED (Abdelali et al., 2014)) as well as corpora developed at the Center for Indian Lan- guage Technology2 (CFILT), IIT Bombay over the years. The training corpus consists of sentences, phrases as well 1work done at IIT Bombay 2www.cfilt.iitb.ac.in as dictionary entries, spanning many applications and do- mains. 2.1. Corpus Details The details of the training corpus are shown in Table 1. We briefly describe the new sub-corpora we have added …

Content truncated to 500 words