Case markers and Morphology: Addressing the crux of the fluency problem in English-Hindi SMT

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 800–808, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Case markers and Morphology: Addressing the crux of the fluency problem in English-Hindi SMT Ananthakrishnan Ramanathan, Hansraj Choudhary Avishek Ghosh, Pushpak Bhattacharyya Department of Computer Science and Engineering Indian Institute of Technology Bombay Powai, Mumbai-400076 India {anand, hansraj, avis, pb}@cse.iitb.ac.in Abstract We report in this paper our work on accurately generating case markers and suffixes in English-to-Hindi SMT. Hindi is a relatively free word-order language, and makes use of a comparatively richer set of case markers and morphological suffixes for correct meaning representa- tion. From our experience of large-scale English-Hindi MT, we are convinced that fluency and fidelity in the Hindi output get an order of magnitude facelift if accurate case markers and suffixes are produced. Now, the moot question is: what entity on the English side encodes the information contained in case markers and suffixes on the Hindi side? Our studies of correspon- dences in the two languages show that case markers and suffixes in Hindi are predom- inantly determined by the combination of suffixes and semantic relations on the En- glish side. We, therefore, augment the aligned corpus of the two languages, with the correspondence of English suffixes and semantic relations with Hindi suffixes and case markers. Our results on 400 test sentences, translated using an SMT sys- tem trained on around 13000 parallel sen- tences, show that suffix + semantic rela- tion → case marker/suffix is a very useful translation factor, in the sense of making a significant difference to output quality as indicated by subjective evaluation as well as BLEU scores. 1 Introduction Two fundamental problems in applying statistical machine translation (SMT) techniques to English- Hindi (and generally to Indian language) MT are: i) the wide syntactic divergence between the lan- guage pairs, and ii) the richer morphology and case marking of Hindi compared to English. The first problem manifests itself in poor word-order in the output translations, while the second one leads to incorrect inflections (word-endings) and case marking. Being a free word-order language, Hindi suffers badly when morphology and case markers are incorrect. To solve the former, word-order related, prob- lem, we use a preprocessing technique, which we have discussed in (Ananthakrishnan et al., 2008). This procedure is similar to what is suggested in (Collins et al., 2005) and (Wang, 2007), and re- sults in the input sentence being reordered to fol- low Hindi structure. The focus of this paper, however, is on the thorny problem of generating case markers and morphology. It is recognized that translating from poor to rich morphology is a challenge (Avramidis and Koehn, 2008) that calls for deeper linguistic analysis to be part of the translation process. Such analysis is facilitated by factored models (Koehn et al., 2007), which provide a framework for incor- porating lemmas, suffixes, POS tags, and any other linguistic factors in a log-linear model for phrase- based SMT. …

Content truncated to 500 words