Building a Word Segmenter for Sanskrit Overnight

Building a Word Segmenter for Sanskrit Overnight Vikas Reddy*1, Amrith Krishna*2, Vishnu Dutt Sharma3, Prateek Gupta4, Vineeth M R5, Pawan Goyal2 1Dept. of Mining Engineering, 2Dept. of Computer Science and Engineering, 4Dept. of Mathematics, 5Dept. of Electrical Engineering, 1,2,4,5IIT Kharagpur 3American Express India Pvt Ltd {vikas.challaram, amrith}@iitkgp.ac.in, pawang@cse.iitkgp.ernet.in Abstract There is abundance of digitised texts available in Sanskrit. However, the word segmentation task in such texts are challenging due to the issue of Sandhi. In Sandhi, words in a sentence often fuse together to form a single chunk of text, where the word delimiter vanishes and sounds at the word boundaries undergo transformations, which is also reflected in the written text. Here, we propose an approach that uses a deep sequence to sequence (seq2seq) model that takes only the sandhied string as the input and predicts the unsandhied string. The state of the art models are linguistically involved and have external dependencies for the lexical and morphological analysis of the input. Our model can be trained “overnight” and be used for production. In spite of the knowledge lean approach, our system preforms better than the current state of the art by gaining a percentage increase of 16.79 % than the current state of the art. Keywords: Word Segmentation, Sanskrit, Low-Resource Languages, Sequence to sequence, seq2seq, Deep Learning 1. Introduction Sanskrit had profound influence as the knowledge preserv- ing language for centuries in India. The tradition of learn- ing and teaching Sanskrit, though limited, still exists in In- dia. There have been tremendous advancements in digitisa- tion of ancient manuscripts in Sanskrit in the last decade. Numerous initiatives such as the Digital Corpus of San- skrit1, GRETIL2, The Sanskrit Library3 and others from the Sanskrit Linguistic and Computational Linguistic commu- nity is a fine example of such efforts (Goyal et al., 2012; Krishna et al., 2017). The digitisation efforts have made the Sanskrit manuscripts easily available in the public domain. However, the accessi- bility of such digitised manuscripts is still limited. Numer- ous technical challenges in indexing and retrieval of such resources in a digital repository arise due to the linguistic peculiarities posed by the language. Word Segmentation in Sanskrit is an important yet non-trivial prerequisite for facilitating efficient processing of Sanskrit texts. Sanskrit has been primarily communicated orally. Due to its oral tradition, the phonemes in Sanskrit undergo euphonic as- similation in spoken format. This gets reflected in writing as well and leads to the phenomena of Sandhi (Goyal and Huet, 2016). Sandhi leads to phonetic transformations at word boundaries of a written chunk, and the sounds at the end of a word join together to form a single chunk of char- acter sequence. This not only makes the word boundaries indistinguishable, but transformations occur to the charac- ters at the word boundaries. The transformations can be deletion, insertion or substitution of one or more sounds at the word ends. There are about 281 sandhi rules, each de- noting a unique combination of phonetic transformations, documented in the grammatical tradition …

Content truncated to 500 words