ThaiLMCut: Unsupervised Pretraining for Thai Word Segmentation

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6947–6957 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6947 ThaiLMCut: Unsupervised Pretraining for Thai Word Segmentation Suteera Seeha1,2, Ivan Bilan2, Liliana Mamani Sanchez2, Johannes Huber2, Michael Matuschek2, Hinrich Schütze1 1Center for Information and Language Processing (CIS), Ludwig Maximilian University of Munich, Germany 2TrustYou GmbH, Munich, Germany suteera.seeha@gmail.com {ivan.bilan, liliana.sanchez, johannes.huber, michael.matuschek}@trustyou.com Abstract We propose ThaiLMCut, a semi-supervised approach for Thai word segmentation which utilizes a bi-directional character language model (LM) as a way to leverage useful linguistic knowledge from unlabeled data. After the language model is trained on substantial unlabeled corpora, the weights of its embedding and recurrent layers are transferred to a supervised word segmentation model which continues fine-tuning them on a word segmentation task. Our experimental results demonstrate that applying the LM always leads to a performance gain, especially when the amount of labeled data is small. In such cases, the F1 Score increased by up to 2.02%. Even on a big labeled dataset, a small improvement gain can still be obtained. The approach has also shown to be very beneficial for out-of-domain settings with a gain in F1 Score of up to 3.13%. Finally, we show that ThaiLMCut can outperform other open source state-of-the-art models achieving an F1 Score of 98.78% on the standard benchmark, InterBEST2009. Keywords: word segmentation, Thai word segmentation, Thai tokenizer, semi-supervised, character language model, pretrained language model 1. Introduction Word segmentation or tokenization is the task of splitting texts into word units. It is an important building block for many Natural Language Processing (NLP) tasks such as Text Classification, Named Entity Recognition (NER), and Machine Translation. Incorrect tokenization leads to misinterpretation of the input text which could potentially affect the performance of the downstream tasks. Tokenizing Thai text is especially difficult because words are written continuously without word delimiters. Spaces can be used in most cases to identify word boundaries in e.g. English or German texts, but this is not the case for Thai and some other Asian languages like Chinese, Japanese, or Vietnamese. In Thai, spaces are used to separate sentences. However, they are used for other purposes as well, such as separating phrases, clauses, and listed items. In practice, the use of spaces in Thai is rather arbitrary due to the nature of the Thai language which allows for a lot of flexibility. State-of-the-art supervised word segmentation systems for Thai report reaching a performance between 97% and 99% F1 Score (Nararatwong et al., 2018; Jousimo et al., 2017; Kittinaradorn et al., 2019; Phuriphatwatthana, 2017; Kongyoung et al., 2015). However, some studies suggest these models might not be able to handle non-standard texts efficiently. Ronran et al. (2016) found that the Thai Lexeme Analyser (TLex) (Haruechaiyasak and Kongyoung, 2009), a tokenizer based on ConditionalRandom Fields (CRFs)(Laf- ferty et al., 2001), was not able to properly segment Twitter1 posts. Lertpiya et al. (2018) revealed that Sertis (Jousimo et al., 2017), a model based on …

Content truncated to 500 words