The LTRC Hindi-Telugu Parallel Corpus

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3417–3424 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3417 The LTRC Hindi-Telugu Parallel Corpus Vandan Mujadia, Dipti Misra Sharma MT-NLP Lab, LTRC, KCIS, IIIT-Hyderabad, India {vandan.mu@research., dipti@}iiit.ac.in Abstract We present the Hindi-Telugu Parallel Corpus of different technical domains such as Natural Science, Computer Science, Law and Healthcare along with the General domain. The qualitative corpus consists of 700K parallel sentences of which 535K sentences were created using multiple methods such as extract, align and review of Hindi-Telugu corpora, end-to-end human translation, iterative back-translation driven post-editing and around 165K parallel sentences were collected from available sources in the public domain. We present the comparative assessment of created parallel corpora for representativeness and diversity. The corpus has been pre-processed for machine translation, and we trained a neural machine translation system using it and report state-of-the-art baseline results on the developed development set over multiple domains and on available benchmarks. With this, we define a new task on Domain Machine Translation for low resource language pairs such as Hindi and Telugu. The developed corpus (535K) is freely available for non-commercial research and to the best of our knowledge, this is the well curated, largest, publicly available domain parallel corpus for Hindi-Telugu. Keywords: Machine Translation, Domain Machine Translation, Parallel Corpora, Indian Languages, Hindi, Telugu 1. Introduction Generally, in multilingual society such as India, people use one language for ethnic identity, another for busi- ness transactions, another for official dealings and yet another for entertainment, rituals and so on(Annamalai, 2001). The language used by a group (eg. Dakkhini1, a variety of Hindi, spoken in the Deccan region of In- dia) living in among another group (e.g. Telugu) is an illustrative case for linguistic convergence(Vasanta et al., 2010). According to the Census of India of 20112, Hindi3 is one of the major languages spoken primarily in the Indian subcontinent. It has around 528 million native speakers and around 700 million total speakers. It is the official language of many North and Central states in India and acts as lingua franca for many In- dians. In contrast, Telugu, the largest member of the Dravidian language family, is the official language4 of the states of Telangana and Andhra Pradesh. It has 95 million native speakers in India and around the world. Hindi and Telugu both are part of 22 official languages recognized by India. For such a diverse group of lan- guage users, in order to have an appropriate communi- cation or to provide educational content in multiple lan- guages, such as Hindi and Telugu, the need for transla- tion systems to translate content in the respective lan- guages becomes essential. Hence, there is immense potential for Hindi-Telugu machine translation and yet, the qualitative parallel cor- pora available between these two languages are limited 1https://en.wikipedia.org/wiki/ Deccani_language 2https://censusindia.gov.in/ 2011Census/pes/Pesreport.pdf 3https://en.wikipedia.org/wiki/Hindi 4https://en.wikipedia.org/wiki/Telugu_ language and specially for the domains. The non-availability of domain parallel data is a major issue in the training …

Content truncated to 500 words