Statistical Sandhi Splitter and its Effect on NLP Applications

Proceedings of Recent Advances in Natural Language Processing, pages 313–319, Hissar, Bulgaria, Sep 7–9 2015. Statistical Sandhi Splitter and its Effect on NLP Applications Prathyusha Kuncham Prathyusha.kuncham@gmail.com IIIT-Hyderabad Kovida Nelakuditi nelakuditi.kovida@research.iiit.ac.in IIIT-Hyderabad Radhika Mamidi radhika.mamidi@iiit.ac.in IIIT-Hyderabad Abstract This paper revisits the work of (Kuncham et al., 2015) which developed a statistical sandhi splitter (SSS) for agglutinative languages that was tested for Telugu and Malayalam lan- guages. Handling compound words is a major challenge for Natural Language Processing (NLP) applications for agglutinative lan- guages. Hence, in this paper we concentrate on testing the effect of SSS on the NLP applica- tions like Machine Translation, Dialogue Sys- tem and Anaphora Resolution and show that the accuracy of these applications is consist- ently improved by using SSS. We shall also discuss in detail the performance of SSS on these applications. 1 Introduction Sandhi which has its origin from Sanskrit ‘samdhi’ meaning “combination”, it refers to a set of morphophonological changes i.e. fusion of final and initial sounds/characters at either mor- pheme or word boundaries. Sandhi is of two types; (i) Internal sandhi and (ii) External sandhi. “Macdonell, 1926” Internal Sandhi: It refers to morphophonologi- cal changes that occur within a word i.e. across morpheme boundaries. For example, consider an English word “impatient” where /n/ in the nega- tive morpheme “in-” has changed to /m/. This is seen for all the words starting with bilabial sounds that are prefixed with the “in-” mor- pheme. External Sandhi: It refers to morphophonologi- cal changes that occur across word boundaries. When different words combine to form a com- pound word, we call it external sandhi. This type of sandhi occurs predominantly in Italian “Absa- lom et al., 2006” and Dravidian languages. Ex1: ‘pUjayyAkA’ -> ‘pUja’+‘ayyAkA’ 1 All the examples are from Telugu language. after having finished the prayer prayer finished Here, we can observe the morphophonological change (a -> a + a) at the word boundaries. “Kuncham et al., (2015) handled one type of external sandhi that is prominent in many agglu- tinative languages and poses many challenges to NLP applications as seen below.” Machine Translation akkaDikeLLu -> akkaDiki + veLLu go there there go When ‘akkaDikeLLu’ is given as input to Google “Telugu to English” Translate, it could not analyze the input and the English translation was ‘Akkadikellu’ which is wrong. But if the compound word is split and then given as input, Google Translate gave correct translation as ‘Go there’. Anaphora Resolution nEnoccAnu -> nEnu + vaccAnu I came I came For the proper functioning of the Anaphora Res- olution system, it is important to identify the pronoun ‘nEnu’ (I) in the input which is only possible by splitting the input. Dialogue System raMgEmiTi -> raMgu + EmiTi what is the color color what If the compound word is given as input to a Dia- logue system, it is very important to identify the question word ‘EmiTi’ (what) to give the correct answer. From the above examples, we can see the im- portance of sandhi …

Content truncated to 500 words