S3 - Statistical Sandhi Splitting

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 301–308, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP S3 - Statistical Sam. dhi Splitting Abhiram Natarajan and Eugene Charniak Brown Laboratory for Linguistic Information Processing Brown University, USA abhiram.nat@gmail.com, ec@cs.brown.edu Abstract The problem of Sam. dhi-Splitting is cen- tral to computational processing of San- skrit texts. Currently the best-known al- gorithm for this task, given a chunk, gen- erates all possible splits and chooses the Maximum-a-Posteriori estimate as the fi- nal answer. Our contributions to the task of Sam. dhi-Splitting are two-fold. Firstly, we improve upon the current algorithm by proposing a principled modification of the posterior probability function to achieve better results. Secondly, we pro- pose an algorithm based on Bayesian Word-Segmentation methods. We find that the unsupervised version of our algorithm achieves a better precision than the cur- rent algorithm with the original probabilis- tic model. We then present a supervised version of our algorithm that outperforms all previous methods/models. 1 Introduction Sanskrit, considered to be among the oldest lan- guages in the world, is elaborate in its oral spec- ifications. It possesses a set of euphonic rules called Sam. dhi1 rules, which when applied, cause phonological changes at word or morph bound- aries. These rules are enumerated in P¯an.ini’s As.t.¯adhy¯ay¯i, and have been devised with the aim of achieving epigrammatic brevity. There are two kinds of Sam. dhi: 1. Internal Sam. dhi: Euphonic transformation at morph boundaries; W1x + yW2 ⇒ W1zW2, where x and y are the final and initial segments of W1 and W2, respectively, and z respresents the smoothed phonetic transfor- mation of x and y together 1Literally means “Putting Together”. E.g: d¯ipena udvejayati ⇒ d¯ipenodvejayati 2. External Sam. dhi: Phonetic change at word boundaries; W1x + W2 ⇒ W1x′ + W2, where the final segment of the first word x changes to x′ E.g: utthitah. vidy¯adharah. ⇒ utthito vidy¯adharah. We shall refer to the process of making the eu- phonic transformations as Sam. dhising and the text formed as a result as Sam. dhied text. The process of undoing the euphonic transformations shall be referred to as Analysis or Sam. dhi-Splitting2, and the text formed as a result will be referred to as Analysed text. It is easy to see that the task of Sam. dhi- Splitting is ambiguous. For instance, any of {(ca,api), (c¯a,api), (ca,¯api), (c¯a,¯api)} could have combined to form c¯api. Contextual knowl- edge is necessary for perfect analysis of Sanskrit text (Hellwig, 2009), and current methods have not evolved enough to be able to supply context. However the field of Statistical Machine Learning, by making reasonable assumptions, allows us to provide approximations of these processes. Pre- vious work (Beesley, 1998; Hyman, 2007) indi- cates that Finite State Transducers (FST) could be used to generate morphologically valid splits. Mit- tal (2010) considers the FST approach as well as an approach based on Optimality Theory (Prince and Smolensky, 1993), by defining a posterior probability function to choose …

Content truncated to 500 words