SemEval-2012 Task 4: Evaluating Chinese Word Similarity

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 374–377, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics SemEval-2012 Task 4: Evaluating Chinese Word Similarity Peng Jin Yunfang Wu School of Computer Science Institute of Computational Linguistics Leshan Normal University Peking University Leshan, 614000, China Beijing, 100871, China jandp@pku.edu.cn wuyf@pku.edu.cn Abstract This task focuses on evaluating word similari- ty computation in Chinese. We follow the way of Finkelstein et al. (2002) to select word pairs. Then we organize twenty under- graduates who are major in Chinese linguis- tics to annotate the data. Each pair is assigned a similarity score by each annotator. We rank the word pairs by the average value of similar scores among the twenty annotators. This data is used as gold standard. Four systems partici- pating in this task return their results. We evaluate their results on gold standard data in term of Kendall's tau value, and the results show three of them have a positive correlation with the rank manually created while the taus' value is very small. 1 Introduction The goal of word similarity is to compute the simi- larity degree between words. It is widely used in natural language processing to alleviate data sparseness which is an open problem in this field. Many research have focus on English language (Lin, 1998; Curran and Moens, 2003; Dinu and Lapata, 2010), some of which rely on the manual created thesaurus such as WordNet (Budanitsky and Hirst, 2006), some of which obtain the similar- ity of the words via large scale corpus (Lee, 1999), and some research integrate both thesaurus and corpus (Fujii et al., 1997). This task tries to evalu- ate the approach on word similarity for Chinese language. To the best of our knowledge, this is first release of benchmark data for this study. In English language, there are two data sets: Ru- benstein and Goodenough (1965) and Finkelstein et al. (2002) created a ranking of word pairs as the benchmark data. Both of them are manually anno- tated. In this task, we follow the way to create the data and annotate the similarity score between word pairs by twenty Chinese native speakers. Finkelstein et al. (2002) carried out a psycholin- guistic experiment: they selected out 353 word pairs, then ask the annotators assign a numerical similarity score between 0 and 10 (0 denotes that words are totally unrelated, 10 denotes that words are VERY closely related) to each pair. By defini- tion, the similarity of the word to itself should be 10. A fractional score is allowed. It should be noted that besides the rank of word pairs, the thesaurus such as Roget's thesaurus are often used for word similarity study (Gorman and Curran, 2006). The paper is organized as follows. In section 2 we describe in detail the process of the data prepa- ration. Section 3 introduces the four participating systems. Section 4 reports their results and gives a brief discussion.. And finally in section 5 we bring forward some …

Content truncated to 500 words