No more beating about the bush : A Step towards Idiom Handling for Indian Language NLP

No more beating about the bush: A Step towards Idiom Handling for Indian Language NLP Ruchit Agrawal, Vighnesh Chenthil Kumar, Vigneshwaran Muralidharan, Dipti Sharma FBK Trento, IIIT Hyderabad, Cardiff University, IIIT Hyderabad ragrawal@fbk.eu, vighnesh.ck@students.iiit.ac.in, muralidaranV@cardiff.ac.uk, dipti@iiit.ac.in Abstract One of the major challenges in the field of Natural Language Processing (NLP) is the handling of idioms; seemingly ordinary phrases which could be further conjugated or even spread across the sentence to fit the context. Since idioms are a part of natural language, the ability to tackle them brings us closer to creating efficient NLP tools. This paper presents a multilingual parallel idiom dataset for seven Indian languages in addition to English and demonstrates its usefulness for two NLP applications - Machine Translation and Sentiment Analysis. We observe significant improvement for both the subtasks over baseline models trained without employing the idiom dataset. Keywords: Idioms, Machine Translation, Sentiment Analysis, Indian Languages 1. Introduction Idioms pose a problem to most NLP applications (Sag et al., 2002), including sentiment analysis, question an- swering, machine translation, parsing and so on. One of the most negatively affected subtasks among these is Machine Translation (MT) (Salton et al., 2014a). While parallel corpora can be used by MT systems to learn the language constructs, thereby generating de- cent translations from source to target language; the same cannot be said for the learning of idioms. Most machine translation systems existent today fail when it comes to the handling of idioms (Table 1). Past re- search (Salton et al., 2014a) has come up with results stating that a standard Statistical Machine Transla- tion (SMT) system tends to achieve only about half the BLEU score of the same system when applied to sentences containing idioms, as compared to those that do not. Since idioms encode a very specific kind of linguis- tic knowledge, it is not easy to learn their automatic handling computationally, without an idiom database. This makes idiom handling a challenging problem for various NLP subtasks including sentiment analysis and question answering in addition to MT. The situation is far worse for Indian languages, a majority of which are low resource languages (Post et al., 2012) with re- gard to the availability of NLP tools, and yet repre- senting 1.3 billion native speakers. Moreover, these languages are under-studied, while also exhibiting lin- guistic properties that make idiom handling for various NLP subtasks even more challenging. In this paper, we present a multilingual parallel dataset that maps 2208 commonly used idioms in English to their translations in seven Indian languages: Hindi, Urdu, Bengali, Tamil, Gujarati, Malayalam and Tel- ugu1. The idioms are also annotated with the appro- priate sentiments that they channel, and their mean- ings in the respective languages. We demonstrate the enhancement obtained using our resource for two ma- 1This dataset is available at goo.gl/receLs jor applications - machine translation and sentiment analysis. We observe a significant improvement in per- formance on conducting baseline experiments for the above mentioned tasks. 2. Related Work One of the earliest known …

Content truncated to 500 words