A Word Labeling Approach to Thai Sentence Boundary Detection and POS Tagging

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 319–327, Osaka, Japan, December 11-17 2016. A Word Labelling Approach to Thai Sentence Boundary Detection and POS Tagging Nina Zhou Institute for Infocomm Research zhoun@i2r.a-star.edu.sg Wang Xuangcong Institute for Infocomm Research wangxc@i2r.a-star.edu.sg Aw AiTi Institute for Infocomm Research aaiti@i2r.a-star.edu.sg Nattadaporn Lertcheva Institute for Infocomm Research lertchevan@i2r.a-star.edu.sg Abstract Previous studies on Thai Sentence Boundary Detection (SBD) mostly assumed a sentence ends at a space and formulated the task SBD as a disambiguation problem, which classified a space either as an indicator for Sentence Boundary (SB) or non-Sentence Boundary (nSB). In this paper, we propose a word labelling approach which treats the space character as a normal word, and detects SB between any two words. This removes the restriction for SB to be occurred only at spaces and makes our system more robust for modern Thai writing. It is because in modern Thai writing, the space is not consistently used to indicate SB. As syntactic information contributes to better SBD, we further propose a joint Part- Of-Speech (POS) tagging and SBD framework based on Factorial Conditional Random Field (FCRF) model. We compare the performance of our proposed approach with reported methods on ORCHID corpus. We also performed experiments of FCRF model on the TaLAPi corpus. The results show that the word labelling approach has better performance than previous space-based classification approaches and FCRF joint model outperforms LCRF model in terms of SBD in all experiments. 1 Introduction Sentence Boundary Detection (SBD) is a fundamental task for many Natural Language Processing (NLP) and analysis tasks, including POS tagging, syntactic, semantic, and discourse parsing, parallel text alignment, and machine translation (Gillick, 2009). Most research on SBD focus on languages that already have a well-defined concept of what a sentence is, typically indicated by sentence-end markers like full-stops, question marks, or other punctuations. However, as we study more contexts of language use (e.g. speech output which lacks punctuations) as well as look at many more different languages, the assumption of clearly-punctuated sentence boundary becomes less valid. One such lan- guage is Thai. In prior research on Thai, the space character has been regarded as a very important element in Thai SBD (Pradit et al., 2000; Paisarn et al., 2001; Glenn et al., 2010). These regard that space characters are always present between sentences. However, in actual fact, as prescribed by Thai linguistic author- ities (www.royin.go.th) as well as what can be observed in real texts, spaces do exist in Thai texts not only in such sentence-end contexts. There is some pressure from linguistic authorities (Wathabun- ditkul, 2003) to set orthographic standards in Thai, prescribing the use of spaces in the context of cer- tain words, following the rules of the Thai Royal Institute Dictionary1. Examples of these rules in- clude: using of the space before and after an interjection or an onomatopoeiac word โอ๊ย (Ouch!), อุ๊ย (ow!); before conjunctions และ (and), หรือ (or), and แต่ (but); before and after a numeric …

Content truncated to 500 words