Translation Quality Estimation for Indian Languages

Translation Quality Estimation for Indian Languages Nisarg Jhaveri Manish Gupta∗ Vasudeva Varma International Institute of Information Technology, Gachibowli, Hyderabad, Telangana - 500 032, India. nisarg.jhaveri@research.iiit.ac.in, {manish.gupta, vv}@iiit.ac.in Abstract Translation Quality Estimation (QE) aims to estimate the quality of an automated machine translation (MT) output without any human intervention or reference trans- lation. With the increasing use of MT systems in various cross-lingual applica- tions, the need and applicability of QE systems is increasing. We study exist- ing approaches and propose multiple neu- ral network approaches for sentence-level QE, with a focus on MT outputs in In- dian languages. For this, we also intro- duce five new datasets for four language pairs: two for English–Gujarati, and one each for English–Hindi, English–Telugu and English–Bengali, which includes one manually post-edited dataset for English– Gujarati. These Indian languages are spo- ken by around 689M speakers world-wide. We compare results obtained using our proposed models with multiple state-of- the-art systems including the winning sys- tem in the WMT17 shared task on QE and show that our proposed neural model which combines the discriminative power of carefully chosen features with Siamese Convolutional Neural Networks (CNNs) works best for all Indian language datasets. 1 Introduction In recent years, Machine Translation (MT) sys- tems have seen significant improvements. How- ever, the quality of the output obtained from these c⃝ 2018 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. ∗The author is also a Principal Applied Scientist at Microsoft. MT systems is neither perfect nor consistent across multiple test cases. The task of Translation Quality Estimation (QE) aims to estimate the quality of an MT output without any reference translation. QE is now critically important with the increas- ing deployment of MT systems in practical envi- ronments. QE has been shown to be extremely use- ful and is widely used in Computer Aided Trans- lation (CAT) environments (Escart´ın et al., 2017; Turchi et al., 2015). QE can also be useful in various applications and systems such as cross- lingual summarization, cross-lingual information retrieval, etc., which rely on high quality transla- tions. With the help of QE, such systems can au- tomatically pick the best translation out of several proposed translations by multiple MT systems. If the estimated quality is still unsatisfactory the sys- tem can alert the user about the poor quality, or fall-back to some alternate way to find a better translation. Word, phrase, sentence or document level QE has been studied extensively by various re- searchers. WMT12-17 (the 7th to 10th workshops on statistical machine translation and the 1st and 2nd conferences on machine translation) held a shared task on QE (Callison-Burch et al., 2012; Bojar et al., 2013, 2014, 2015, 2016, 2017). The shared task has explored QE on several datasets and settings for English–Spanish and English– German language pairs over years. Little work has been done to study QE for In- dian languages. In this work, we focus on four In- dian languages: Telugu, Hindi, …

Content truncated to 500 words