Extract Chinese Unknown Words from a Large-scale Corpus Using Morphological and Distributional Evidences

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 837–845, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Extract Chinese Unknown Words from a Large-scale Corpus Using Morphological and Distributional Evidences Kaixu Zhang† and Ruining Wang† and Ping Xue‡ and Maosong Sun† †State Key Laboratory on Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology Department of Computer Science and Technology Tsinghua University, Beijing 100084, China {kareyzhang,wangruining.student,sunmaosong}@gmail.com ‡The Boeing Company ping.xue@boeing.com Abstract The representative method of using mor- phological evidence for Chinese unknown word (UW) extraction is Chinese word segmentation (CWS) model, and the method of using distributional evidence for UW extraction is accessor variety (AV) criterion. However, neither of these meth- ods has been verified on large-scale cor- pus. In this paper, we propose exten- sions to remedy the drawbacks of these two methods to handle large-scale corpus: (1) for CWS, we propose a generalized definition of word to improve the recall; and (2) for AV, we propose a restricted ver- sion to decrease noise. We carry out ex- periments on a Chinese Web corpus with approximate 200 billion Chinese charac- ters. Experimental results show that our methods outperform the baselines, and the combination of the two evidences can fur- ther improve the performance. Moreover, our methods can also efficiently segment the corpus on the fly, which is especially valuable for processing large-scale corpus. 1 Introduction A Chinese word is constructed with one or more Chinese characters. Chinese characters can am- biguously combine to form Chinese words, and there are no explicit delimiters in the text to in- dicate word boundaries. It is thus crucial for most Chinese natural language processing tasks to maintain a large word list. Given that Chinese lan- guage has several productive word creation mech- anisms, identification and extraction of UW is an important task for or Chinese NLP tasks. Chinese unknown word (UW) extraction aims to extract UWs from a given corpus and enrich the word list. Two types of information can be used to determine whether a string of characters in question is a Chinese UW or not, namely the characters that construct the string in question, and the neighbors that this string of characters appears with. The first type of information can be regarded as the morphological evidence, while the second can be viewed as the distributional evidence. The representative method of using morpho- logical evidence is Chinese word segmentation (CWS) model. CWS is to identify every word to- ken in a given sentence. Using the CWS model, we can define the word-string ratio (WSR) to ex- tract UWs. The representative method of using distributional evidence is the accessor variety (AV) criterion (Feng et al., 2004a). WSR is directly derived from the CWS method based on character tagging (Xue, 2003). This CWS method is based on the morphological infor- mation of the strings in question and their context. Strings with high WSR are considered as words, for high WSR indicates that the corresponding string …

Content truncated to 500 words