How does Dictionary Size Influence Performance of Vietnamese Word Segmentation?

How does Dictionary Size Influence Performance of Vietnamese Word Segmentation? Wuying Liu, Lin Wang Laboratory of Language Engineering and Computing, Guangdong University of Foreign Studies 510420 Guangzhou, Guangdong, CHINA E-mail: wyliu@gdufs.edu.cn, wanglin@nudt.edu.cn Abstract Vietnamese word segmentation (VWS) is a challenging basic issue for natural language processing. This paper addresses the problem of how does dictionary size influence VWS performance, proposes two novel measures: square overlap ratio (SOR) and relaxed square overlap ratio (RSOR), and validates their effectiveness. The SOR measure is the product of dictionary overlap ratio and corpus overlap ratio, and the RSOR measure is the relaxed version of SOR measure under an unsupervised condition. The two measures both indicate the suitable degree between segmentation dictionary and object corpus waiting for segmentation. The experimental results show that the more suitable, neither smaller nor larger, dictionary size is better to achieve the state-of-the-art performance for dictionary-based Vietnamese word segmenters. Keywords: Vietnamese Word Segmentation, Dictionary-based Segmentation Algorithm, Dictionary Size, Square Overlap Ratio, Relaxed Square Overlap Ratio 1. Introduction Like Thai, Japanese and Chinese text, Vietnamese text is also a text without any explicit separator between words. Thus, identifying the word boundaries is a challenging basic issue to above oriental languages for natural language processing (Doan, 2008). Vietnamese is a monosyllabic language, whose basic linguistic unit is called 'tiếng', similar to traditional syllables in respect of phonetic form. A Vietnamese word can be made up of a single syllable, or several sequential syllables connected by space symbols. In raw Vietnamese texts, space symbol can be treated as an overload symbol, which is a connector within a word or is a separator between words. Therefore, the Vietnamese word segmentation (VWS) problem can be defined as a binary categorization task for each space symbol. If a space symbol is a connector in a word, we will output a symbol ('_') to replace it. And if a space symbol is a separator between words, we will maintain it as a space symbol (' ') in the segmented result. Since the early days of Vietnamese information processing researches, VWS problem has been widely investigated, and many effective segmentation algorithms have been proposed (Dinh et al., 2008). The early dictionary-based word segmentation algorithms mainly include maximum matching algorithm and reverse maximum matching algorithm. Subsequently, various kinds of advanced machine learning algorithms (such as maximum entropy (Dinh and Vu, 2006), support vector machines and conditional random fields (Nguyen et al., 2006)) regard word segmentation problem as a sequence labeling task, and related algorithms can obtain preferable performance in VWS. Some investigations show that many affixed resources (such as part of speech tag (Pham et al., 2009)) are helpful in the word segmentation algorithm (Tran et al., 2010). Recently, hybrid and ensemble algorithms have attracted more and more attentions. The hybrid algorithm (Le et al., 2008) combines finite-state automata, regular expression and maximum matching techniques to implement a highly accurate Vietnamese tokenizer (vnTokenizer). The ensemble algorithm (Liu and Lin, 2014) combines multiple weak segmenters to form a strong Vietnamese …

Content truncated to 500 words