Vietnamese Word Segmentation with CRFs and SVMs: An Investigation

Vietnamese Word Segmentation with CRFs and SVMs: An Investigation Cam-Tu Nguyen1, Trung-Kien Nguyen1, Xuan-Hieu Phan2 Le-Minh Nguyen2, and Quang-Thuy Ha1 1 College of Technology, Vietnam National University, Hanoi 2 School of Information Science, Japan Advanced Institute of Science and Technology ncamtu@vnu.edu.vn , ntkien@vnu.edu.vn , hieuxuan@jaist.ac.jp nguyenml@jaist.ac.jp , thuyhq@vnu.edu.vn Abstract. Word segmentation for Vietnamese, like for most Asian languages, is an important task which has a significant impact on higher language processing levels. However, it has received little attention of the community due to the lack of a common annotated corpus for evaluation and comparison. Also, most previous studies focused on unsupervised-statistical approaches or combined too many techniques. Consequently, their accuracies are not as high as expected. This paper reports a careful investigation of using conditional random fields (CRFs) and support vector machines (SVMs) - two of the most successful statistical learning methods in NLP and pattern recognition - for solving the task. We first build a moderate annotated corpus using different sources of materials. For a careful evaluation, different CRF and SVM models using different feature settings were trained and their results are compared and contrasted with each other. In addition, we discuss several important points about the accuracy, computational cost, corpus size and other aspects that might influence the overall quality of Vietnamese word segmentation. Keywords: Word segmentation, segmenting and labeling sequence data, conditional random fields, support vector machines, maximum matching. 1 Introduction Word segmentation is one of the fundamental preprocessing steps in NLP for building higher applications. It is even more important and challenging in Asian languages, such as Chinese, Japanese, and Korea, because there is no white space between two consecutive words. Vietnamese language faces a similar problem due to the fact that a word may contain more than one separated syllables, and therefore the white space is not always the word separator. In recent years, word boundary detection for Vietnamese has received more attention from the community and there have been several statistical and machine learning methods applied to the task. However, most of the current methods either suffer from unsatisfactory results [1, 2] (with accuracy of 91% or lower) or must combine many techniques in a multi-level processing to obtain good results [3]. In addition, their works were done without comparison to any baseline or study of the quality of the corpus (e.g., the out-of-vocabulary rate, the number of date/time and numbers). Also, there is still no common standard annotated corpus for evaluation and comparison. In this paper, we present a thorough investigation of using two powerful statistical learning methods, CRFs and SVMs, to perform the task. To do so, we first build an annotated corpus of about 8000 sentences with word boundary marked. Although the corpus is not large enough to cover a broad range of Vietnamese vocabularies, it contains documents from different domains to reduce the imbalance in word distribution. Then, CRF and SVM models are trained on the corpus using various feature configurations and their performances are compared and contrasted with each …

Content truncated to 500 words