Combining Prediction by Partial Matching and Logistic Regression for Thai Word Segmentation

Combining Prediction by Partial Matching and Logistic Regression for Thai Word Segmentation Ohm Sornil Department of Computer Science National Institute of Development Administration, Bangkok, Thailand osornil@as.nida.ac.th Paweena Chaiwanarom National Statistical Office Bangkok, Thailand paweena@nso.go.th Abstract Word segmentation is an important part of many applications, including information retrieval, information filtering, document analysis, and text summarization. In Thai language, the process is complicated since words are written continuously, and their structures are not well-defined. A recognized effective approach to word segmentation is Longest Matching, a method based on dictionary. Nevertheless, this method suffers from character-level and syllable-level ambiguities in determining word boundaries. This paper proposes a technique to Thai word segmentation using a two-step approach. First, text is segmented, using an application of Prediction by Partial Matching, into syllables whose structures are more well-defined. This reduces the earlier type of ambiguity. Then, the syllables are combined into words by an application of a syllable-level longest matching method together with a logistic regression model which takes into account contextual information. The experimental results show the syllable segmentation accuracy of more than 96.65% and the overall word segmentation accuracy of 97%. 1 Introduction In Thai language, characters are written without explicit word boundaries. Depending on the contexts, there can be many ways to break a string into words, for instance, "อาจอง" can be segmented as "อา*จอง" or "อาจ*อง", and "นั่งตากลม" can be segmented as "นั่ง*ตา*กลม" or "นั่ง*ตาก*ลม". This complicates the task of identifying word boundaries. Longest matching is the most popular approach to Thai word segmentation (Pooworawan, 1986). The algorithm scans text from left to right and selects the longest match with a dictionary entry at each point, in a greedy fashion. However, longest possible words may not comply with the actual meanings. For example, "ชาวบานรอกราบพระ" segmented by the longest matching as "ชาวบาน-รอก-is ราบ-พระ" instead of the correct segmentation "ชาวบาน- รอ-กราบ-พระ". This type of ambiguity is referred to as character-level ambiguity. In addition, "เขารับรองเทา จากเพื่อน" is segmented as "เขา-รับรอง-เทา-จาก-เพื่อน" instead of the correct segmentation "เขา-รับ-รองเทา-จาก-เพื่อน". This is referred to as syllable-level ambiguity. The technique we propose is a two-step process to word segmentation. In the first step, text is segmented into a sequence of syllables, whose structures are more well-defined. This reduces the character-level ambiguity. The remaining syllable- level ambiguity is the task of combining those syllables into words. 2 Related Work In addition to the longest matching algorithm, discussed earlier, the maximum matching algorithm (Sornlertlamvanich, 1993) was proposed to get around the greedy characteristic of the longest matching algorithm by generating all possible segmentations for a sentence and then selecting the one which contains the fewest number of entries in the dictionary. An application of statistical techniques was proposed by (Pornprasertkul, 1994), using a Viterbi-based approach to exploit statistical information derived from grammatical tags. Later, (Kawtrakul and Chalathip, 1995) and (Meknawin et al., 1997) used variants of the trigram model to compute the most likely segmentation. (Theeramunkong and Sornlertlamvanich, 2000) observed that, in Thai language, some contiguous characters tend to be inseparable units, …

Content truncated to 500 words