Evaluation via Negativa of Chinese Word Segmentation for Information Retrieval

Evaluation via Negativa of Chinese Word Segmentation for Information Retrieval∗ Mike Tian-Jian Jiang ac, Cheng-Wei Shih bc, Chan-Hung Kuo c, Richard Tzong-Han Tsai d, and Wen-Lian Hsu abc aDepartment of Computer Science, National Tsing Hua University, Hsinchu, Taiwan b Institute of Information Systems and Application, National Tsing Hua University, Hsinchu, Taiwan c Institute of Information Science, Academia Sinica, Taipei, Taiwan {tmjanng, dapi, laybow, hsu}@iis.sinica.edu.tw d Department of Computer Science & Engineering, Yuan Ze University, Taoyuan, Taiwan thtsai@saturn.yzu.edu.te Abstract. Numerous studies have analyzed the influences of word segmentation (WS) performance on information retrieval (IR) for Mandarin Chinese and have demonstrated a non-monotonic relationship between WS accuracy and IR effectiveness. The usefulness of the compound words that have been a focus of the IR literature is not reflected by common WS evaluation metrics of word-based precision (P) and recall (R). This investigation proposes alternative measurements of WS accuracy, which are based on negative segments that are annotated against four standards of referenced corpora, called true negative rate (TNR) and negative predictive value (NPV), and compares with P and R through search engine simulation,. Accuracy-controlled WS systems segment queries for the simulation including NTCIR collections and "Sogou" logs. Mean average precision (MAP) estimates the similarity of search results between the original and segmented queries. The statistics demonstrate that TNR and NPV are generally more closely correlated with MAP than are P and R. Keywords: Word segmentation, information retrieval, true negative rate. ∗ This research was supported in part by the National Science Council under grant NSC 100-2631-S-001-001, and the research center for Humanities and Social Sciences under grant IIS-50-23. Ted Kony is appreciated for his editorial assistance. The authors would like to thank anonymous reviewers for their constructive criticisms. Copyright 2011 by Mike Tian-Jian Jiang, Cheng-Wei Shih, Chan-Hung Kuo, Richard Tzong-Han Tsai, and Wen-Lian Hsu 1 Introduction Word segmentation (WS) is an essential preparatory work for Chinese text processing applications and consequently has become a focus of research during the past two decades. Numerous researchers have urged the application of various algorithms to produce “accurate” tokenized text in which the segmentations are performed to match one of the standards of Chinese corpora, such as Academia Sinica Balanced Corpus (Huang et al., 1997) and Chinese Treebank of University of Pennsylvania (Xia, 1999), and evaluated using various metrics, such as word-based precision (P), recall (R) and their harmonic average, termed F1 measure score (F- score), popularized by SIGHAN Chinese WS bakeoffs (Sproat and Emerson, 2003; Emerson, 2005; Levow, 2006; Jin and Chen, 2007; Zhao and Liu, 2010). Subsequently, WS systems that 25th Pacific Asia Conference on Language, Information and Computation, pages 100–109 100 perform strongly in terms of F-score have been considered reliable and are selected as the preprocessors for the commonly accepted notion that WS accuracy is directly related to the effectiveness of follow-up applications. However, this perspective has sometimes failed to win significant support, particularly in the Chinese IR area, because of the absence of strong evidence confirming the causation. As described later …

Content truncated to 500 words