Non-Dictionary-Based Thai Word Segmentation Using Decision Trees

Non-Dictionary-Based Thai Word Segmentation Using Decision Trees Thanaruk Theeramunkong1 Information Technology Program Sirindhorn International Institute of Technology Thammasat University, Pathumthani 12121, Thailand +66-2-986-9103(-8) Ext. 2004 ping@siit.tu.ac.th Sasiporn Usanavasin Information Technology Program Sinrindhorn International Institute of Technology Thammasat University, Pathumthani 12121, Thailand +66-2986-9103(-8) Ext. 2002 sasiporn@kind.siit.tu.ac.th ABSTRACT For languages without word boundary delimiters, dictionaries are needed for segmenting running texts. This figure makes segmentation accuracy depend significantly on the quality of the dictionary used for analysis. If the dictionary is not sufficiently good, it will lead to a great number of unknown or unrecognized words. These unrecognized words certainly reduce segmentation accuracy. To solve such problem, we propose a method based on decision tree models. Without use of a dictionary, specific information, called syntactic attribute, is applied to identify the structure of Thai words. C4.5 is used as a tool for this purpose. Using a Thai corpus, experiment results show that our method outperforms some well-known dictionary-dependent techniques, maximum and longest matching methods, in case of no dictionary. Keywords Decision trees, Word segmentation without a dictionary 1. INTRODUCTION Word segmentation is a crucial topic in analysis of languages without word boundary markers. Many researchers have been trying to develop and implement in order to gain higher accuracy. Unlike in English, word segmentation in Thai, as well as in many other Asian languages, is more complex because the language does not have any explicit word boundary delimiters, such as a space, to separate between each word. It is even more complicated to precisely segment and identify the word boundary in Thai language because there are several levels and several roles in Thai characters that may lead to ambiguity in segmenting the words. In the past, most researchers had implemented Thai word segmentation systems based on using a dictionary ([2], [3], [4], [6], [7]). When using a dictionary, word segmentation has to cope with an unknown word problem. Up to present, it is clear that most researches on Thai word segmentation with a dictionary suffer from this problem and then introduce some particular process to handle such problem. In our preliminary experiment, we extracted words from a pre-segmented corpus to form a dictionary, randomly deleted some words from the dictionary and used the modified dictionary in segmentation process based two well-known techniques; Maximum and Longest Matching methods. The result is shown in Figure 1. The percentages of accuracy with different percentages of unknown words are explored. We found out that in case of no unknown words, the accuracy is around 97% in both maximum matching and longest matching but the accuracy drops to 54% and 48% respectively, in case that 50% of words are unknown words. As the percentage of unknown words rises, the percentage of accuracy drops continuously. This result reflects seriousness of unknown word problem in word segmentation. 1 Accuracy (%) Unknown word (%) Maximum Matching Longest Matching 0 97.24 97.03 5 95.92 95.63 10 93.12 92.23 15 89.99 87.97 20 86.21 82.60 25 78.40 74.41 30 68.07 64.52 35 69.23 …

Content truncated to 500 words