A Fast and Accurate Vietnamese Word Segmenter

A Fast and Accurate Vietnamese Word Segmenter Dat Quoc Nguyen1, Dai Quoc Nguyen2, Thanh Vu3, Mark Dras4, Mark Johnson4 1The University of Melbourne, Australia; 2Deakin University, Australia; 3Newcastle University, United Kingdom; 4Macquarie University, Australia dqnguyen@unimelb.edu.au, dai.nguyen@deakin.edu.au, thanh.vu@newcastle.ac.uk, {mark.dras, mark.johnson}@mq.edu.au Abstract We propose a novel approach to Vietnamese word segmentation. Our approach is based on the Single Classification Ripple Down Rules methodology (Compton and Jansen, 1990), where rules are stored in an exception structure and new rules are only added to correct segmentation errors given by existing rules. Experimental results on the benchmark Vietnamese treebank show that our approach outperforms previous state-of-the-art approaches JVnSegmenter, vnTokenizer, DongDu and UETsegmenter in terms of both accuracy and performance speed. Our code is open-source and available at: https://github.com/datquocnguyen/RDRsegmenter. Keywords: Vietnamese, Word segmentation, Single classification ripple down rules 1. Introduction Word segmentation is referred to as an important first step for Vietnamese NLP tasks (Dien et al., 2001; Ha, 2003; Duc Cong et al., 2016). Unlike English, white space is a weak indicator of word boundaries in Viet- namese because when written, it is also used to sep- arate syllables that constitute words. For example, a written text “thuế thu nhập cá nhân” (individualcá_nhân incomethu_nhập taxthuế) consisting of 5 syllables forms a two-word phrase “thuế_thu_nhập cá_nhân.”1 More specifically, about 85% of Vietnamese word types are composed of at least two syllables and 80%+ of syl- lable types are words by themselves (Thang et al., 2008; Le et al., 2008), thus creating challenges in Viet- namese word segmentation (Nguyen et al., 2012). Many approaches are proposed for the Vietnamese word segmentation task. Le et al. (2008), Pham et al. (2009) and Tran et al. (2012) applied the maxi- mum matching strategy (NanYuan and YanBin, 1991) to generate all possible segmentations for each input sentence; then to select the best segmentation, Le et al. (2008) and Tran et al. (2012) used n-gram language models while Pham et al. (2009) employed part-of- speech (POS) information from an external POS tag- ger. In addition, Nguyen et al. (2006), Dinh and Vu (2006) and Tran et al. (2010) considered this segmen- tation task as a sequence labeling task, using either a linear-chain CRF, SVM or MaxEnt model to assign each syllable a segmentation tag such as B (Begin 1In the traditional underscore-based representation in the Vietnamese word segmentation task (Nguyen et al., 2009), white space is only used to separate words while un- derscore is used to separate syllables inside a word. of a word) or I (Inside of a word). Another promis- ing approach is joint word segmentation and POS tagging (Takahashi and Yamamoto, 2016; Nguyen et al., 2017b), which assigns a combined segmentation and POS tag to each syllable. Furthermore, Luu and Kazuhide (2012), Liu and Lin (2014) and Nguyen and Le (2016) proposed methods based on pointwise pre- diction (Neubig and Mori, 2010), where a binary clas- sifier is trained to identify whether or not there is a word boundary between two syllables. In this paper, we propose …

Content truncated to 500 words