Detecting Sentence Boundaries in Sanskrit Texts

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 288–297, Osaka, Japan, December 11-17 2016. Detecting Sentence Boundaries in Sanskrit Texts Oliver Hellwig Düsseldorf University, SFB 991 ohellwig@phil-fak.uni-duesseldorf.de Abstract The paper applies a deep recurrent neural network to the task of sentence boundary detection in Sanskrit, an important, yet underresourced ancient Indian language. The deep learning approach improves the F scores set by a metrical baseline and by a Conditional Random Field classifier by more than 10%. 1 Introduction Most NLP tasks that deal with written texts take it for granted that sentences are separated reliably by punctuation marks, although punctuation has been added quite late to many writing systems. The large corpora in Old- and Middle-Indian languages, which belong to the central sources for understanding the history of South Asia, generally lack dedicated punctuation marks. This paper applies deep recurrent neural networks (RNN) to a combination of morphological and lexical features for detecting sentence boundaries (SB) in Sanskrit, the oldest and most important of these Indian languages. Traditional editions of Sanskrit texts use a scriptio continua, which lacks several orthographic el- ements that structure texts in modern Western languages. Single words are frequently not separated by blank spaces due to missing orthographic regulation, or because the words are merged through the euphonic rules called sandhi (“connection”).1 Moreover, Sanskrit texts don’t have a consistent and un- ambiguous system for marking SBs. Editors and scribes insert so called (double) dan. d. as (“sticks”, indicated by | and || in this paper) to mark the end of metrical structures. The position of these dan. d. as can be derived directly from the prosodic structure of a text, and dan. d. as always occur at the end of text lines, which coincide with half-verses in most printed editions. While single dan. d. as mark the end of a half-verse, double dan. d. as should, at least theoretically, indicate, where a stanza in the given metre is completed. Double dan. d. as typically occur after every second line or half-verse of a metrical text, be- cause the stanzas are finished at these points. In this function, they are meant to improve the readability of a text. As many sentences terminate at the end of a half-verse or of a stanza, dan. d. as provide a good baseline for punctuation prediction (refer to Table 3). Many editors, however, also insert double dan. d. as after a single or after three metrical lines, when they feel that a sentence is completed at these positions.2 In this way, the purely metrical motivation of double dan. d. as is mingled with the new function of a punc- tuation mark – leaving aside the fact that the philologically interesting inner-line SBs cannot be marked in the dan. d. a system. Linguistic peculiarities of Sanskrit complicate the task. English, for example, encodes a large amount of its syntax through a strongly regulated word order, and structures its sentences by subordinating con- juctions. While these data …

Content truncated to 500 words