Syllable-based Neural Thai Word Segmentation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4619–4637 Barcelona, Spain (Online), December 8-13, 2020 4619 Syllable­based Neural Thai Word Segmentation Pattarawat Chormai∗,⋆,†, Ponrawee Prasertsom‡, Jin Cheevaprawatdomrong‡ and Attapol T. Rutherford‡ ‡Department of Linguistics, Chulalongkorn University, Bangkok, Thailand †Computer Science and Electrical Engineering, Technische Universität, Berlin, Germany ⋆Max Planck School of Cognition, MPI for Human Cognitive and Brain Sciences, Leipzig, Germany p.chormai@tu-berlin.de, {ponrawee.pra, jin.ch}@gmail.com, attapol.t@chula.ac.th Abstract Word segmentation is a challenging pre­processing step for Thai Natural Language Processing due to the lack of explicit word boundaries. The previous systems rely on powerful neural network architecture alone and ignore linguistic substructures of Thai words. We utilize the linguistic observation that Thai strings can be segmented into syllables, which should narrow down the search space for the word boundaries and provide helpful features. Here, we propose a neural Thai Word Segmenter that uses syllable embeddings to capture linguistic constraints and uses dilated CNN filters to capture the environment of each character. Within this goal, we develop the first ML­based Thai orthographical syllable segmenter, which yields syllable embeddings to be used as features by the word segmenter. Our word segmentation system outperforms the previous state­of­the­art system in both speed and accuracy on both in­domain and out­domain datasets. 1 Introduction Word segmentation presents a fundamental challenge for Thai language processing. Many of the down­ stream natural language processing tasks require that texts be broken into a sequence of words before applying any models. For example, bag­of­word models or RNN models usually require that the text is represented as a sequence of words. Notable examples of writing systems without clear word boundaries are Japanese, Chinese, and Khmer, which often require an automatic word segmentation. However, the Thai script differs from Chinese and Japanese in that a Thai character represents just a consonant or a vowel, but a Chinese character represents a whole syllable. The Thai script uses 44 consonant symbols, 15 vowel symbols, and 4 tonal symbols. A word is composed of one or more syllables, and each syllable is formed by a set of intricate orthographical rules for valid sequences of Thai alphabet symbols. Hence, it is not straightforward to extend the techniques from Chinese or Japanese to Thai word segmentation problem. Thai Word segmentation is complicated by linguistic ambiguity and out­of­vocabulary cases. A word can be formed by juxtaposing two “words” e.g. เห็นชอบ (/hˇen tChˆO:p/, approve) = เห็น (/hˇen/, see) + ชอบ (/tChˆO:p/, like). This kind of word formation can be detected with a simple dictionary lookup, but harder cases that require context abound in the language. For example, กอดอกไม้ can be segmented into กอ|ดอก|ไม้ (/kO: d`O:k m´aj/, flower bush) or กอด|อก|ไม้ (/k`O:t P´ok m´aj/, hugging a wooden chest), but the latter is rather nonsensical and statistically unlikely. The local context is needed to select the right segmenta­ tion; therefore, dictionaries only provide partial solutions to this problem. Further, a constant stream of new words and loanwords complicates the task of word segmentation because they never appear in the dictionaries. In …

Content truncated to 500 words