The IIT Bombay Hindi-English Translation System at WMT 2014

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 90–96, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics The IIT Bombay Hindi⇔English Translation System at WMT 2014 Piyush Dungarwal, Rajen Chatterjee, Abhijit Mishra, Anoop Kunchukuttan, Ritesh Shah, Pushpak Bhattacharyya Department of Computer Science and Engineering Indian Institute of Technology, Bombay {piyushdd,rajen,abhijitmishra,anoopk,ritesh,pb}@cse.iitb.ac.in Abstract In this paper, we describe our English- Hindi and Hindi-English statistical sys- tems submitted to the WMT14 shared task. The core components of our translation systems are phrase based (Hindi-English) and factored (English-Hindi) SMT sys- tems. We show that the use of num- ber, case and Tree Adjoining Grammar information as factors helps to improve English-Hindi translation, primarily by generating morphological inflections cor- rectly. We show improvements to the translation systems using pre-procesing and post-processing components. To over- come the structural divergence between English and Hindi, we preorder the source side sentence to conform to the target lan- guage word order. Since parallel cor- pus is limited, many words are not trans- lated. We translate out-of-vocabulary words and transliterate named entities in a post-processing stage. We also investi- gate ranking of translations from multiple systems to select the best translation. 1 Introduction India is a multilingual country with Hindi be- ing the most widely spoken language. Hindi and English act as link languages across the coun- try and languages of official communication for the Union Government. Thus, the importance of English⇔Hindi translation is obvious. Over the last decade, several rule based (Sinha, 1995) , in- terlingua based (Dave et. al., 2001) and statistical methods (Ramanathan et. al., 2008) have been ex- plored for English-Hindi translation. In the WMT 2014 shared task, we undertake the challenge of improving translation between the English and Hindi language pair using Statisti- cal Machine Translation (SMT) techniques. The WMT 2014 shared task has provided a standard- ized test set to evaluate multiple approaches and avails the largest publicly downloadable English- Hindi parallel corpus. Using these resources, we have developed a phrase-based and a factored based system for Hindi-English and English-Hindi translation respectively, with pre-processing and post-processing components to handle structural divergence and morphlogical richness of Hindi. Section 2 describes the issues in Hindi↔English translation. The rest of the paper is organized as follows. Section 3 describes corpus preparation and exper- imental setup. Section 4 and Section 5 describe our English-Hindi and Hindi-English translation systems respectively. Section 6 describes the post- processing operations on the output from the core translation system for handling OOV and named entities, and for reranking outputs from multiple systems. Section 7 mentions the details regarding our systems submitted to WMT shared task. Sec- tion 8 concludes the paper. 2 Problems in Hindi⇔English Translation Languages can be differentiated in terms of structural divergences and morphological mani- festations. English is structurally classified as a Subject-Verb-Object (SVO) language with a poor morphology whereas Hindi is a morpho- logically rich, Subject-Object-Verb (SOV) lan- guage. Largely, these divergences are responsi- ble for the difficulties in translation using a phrase based/factored …

Content truncated to 500 words