Tibetan Word Segmentation as Syllable Tagging Using Conditional Random Field

Tibetan Word Segmentation as Syllable Tagging Using Conditional Random Field ⋆ Huidan Liu a,b, Minghua Nuo a,b, Longlong Ma a, Jian Wu a, and Yeping He a aInstitute of Software, Chinese Academy of Sciences, No.4 South Fourth Street, Zhong Guan Cun, Haidian District, Beijing 100190, China bGraduate University of the Chinese Academy of Sciences, No.80 Zhongguancun East Road, Haidian District, Beijing 100190, China {huidan, minghua, longlong, wujian, yeping}@iscas.ac.cn Abstract. In this paper, we proposed a novel approach for Tibetan word segmentation using the conditional random field. We reformulate the segmentation as a syllable tagging problem. The approach labels each syllable with a word-internal position tag, and combines syllable(s) into words according to their tags. As there is no public available Tibetan word segmenta- tion corpus, the training corpus is generated by another segmenter which has an F-score of 96.94% on the test set. Two feature template sets namely TMPT-6 and TMPT-10 are used and compared, and the result shows that the former is better. Experiments also show that larger training set improves the performance significantly. Trained on a set of 131,903 sentences, the segmenter achieves an F-score of 95.12% on the test set of 1,000 sentences. Keywords: Tibetan word segmentation, Tibetan, syllable tagging, CRF. 1 Introduction During the 7th Century AD, Thume Sambota reputedly devised a script for Tibetan based on the Devanagari model. At present, Tibetan script is used to write both Tibetan language and Dzongkha (Bhutanese) language. The former is a language spoken by about 6 million people in China, India, Bhutan, Sikkim, Ladakh and Nepal, while the later is spoken by about 130,000 people in Bhutan as a national and official language. Just like Chinese, Tibetan text is written without natural word delimiters, so word segmentation is an essential, fundamental and foremost step in Tibetan language processing. Researchers mainly use maximum matching method and some grammar rules to segment Tibetan text at present. In this paper, we reformulate Tibetan word segmentation as a syllable tagging problem, and propose an approach using the conditional random field (CRF) for Tibetan word segmentation. The paper is organized as follows: In Section 2 we recall related work on Tibetan word segmen- tation and Chinese word segmentation methods by character tagging. In Section 3, we simply introduce Tibetan script. We propose the approach in Section 4. Then, in Section 5 we make experiments to evaluate the segmenter, compare it with other Tibetan segmenters. Section 6 con- cludes the paper. 2 Related Work In this section, we first recall the research history and current situation on Tibetan/Dzongkha word segmentation. We also recall research on Chinese word segmentation by character tagging which we have drawn inspiration from. ⋆ The research is partially supported by National Science & Technology Major Project (No.2010ZX01036-001-002, No.2010ZX01036-001-002) and CAS Action Plan for the Development of Western China (No.KGCX2-YW-512). Copyright 2011 by Huidan Liu, Minghua Nuo, Longlong Ma, Jian Wu, and Yeping He 25th Pacific Asia Conference on Language, Information and Computation, pages 168–177 168 2.1 Tibetan/Dzongkha Word Segmentation Chen …

Content truncated to 500 words