EVALution-MAN: A Chinese Dataset for the Training and Evaluation of DSMs

EVALution-MAN: A Chinese Dataset for the Training and Evaluation of DSMs Hongchao Liu, Karl Neergaard, Enrico Santus, Chu-Ren Huang CBS, The Hong Kong Polytechnic University 11 Yuk Choi Rd. Hunghom, Hong Kong jiye12yuran@126.com, {karlneergaard, esantus}@gmail.com, churen.huang@polyu.edu.hk Abstract Distributional semantic models (DSMs) are currently being used in the measurement of word relatedness and word similarity. One shortcoming of DSMs is that they do not provide a principled way to discriminate different semantic relations. Several approaches have been adopted that rely on annotated data either in the training of the model or later in its evaluation. In this paper, we introduce a dataset for training and evaluating DSMs on semantic relations discrimination between words, in Mandarin, Chinese. The construction of the dataset followed EVALution 1.0, which is an English dataset for the training and evaluating of DSMs. The dataset contains 360 relation pairs, distributed in five different semantic relations, including antonymy, synonymy, hypernymy, meronymy and nearsynonymy. All relation pairs were checked manually to estimate their quality. In the 360 word relation pairs, there are 373 relata. They were all extracted and subsequently manually tagged according to their semantic type. The relatas’frequency was calculated in a combined corpus of Sinica and Chinese Gigaword. To the best of our knowledge, EVALution-MAN is the first of its kind for Mandarin, Chinese. Keywords: Dataset, Distributioanl Semantic Models, Tarining, Evaluating 1. Introduction Distributional semantic models (DSMs) have been applied to both the measurement of semantic similarity and relat- edness, and are of paramount importance for tasks such as word sense disambiguation, lexical replacement, dictio- nary construction, and entailment understanding, to name a few (Sun, 2014). DSMs are founded on the assumption that the lexical similarity between words depends on their distributed context. According to the Distributional Hy- pothesis, if compared concepts or words have similar dis- tributional features (hence, context), they are likely to have higher semantic similarity (Harris, 1954). One major short- coming of current DSMs is that they cannot be used for dis- crimination among different relationships between words (Santus et al., 2014c). In recent years, DSMs have been adopted in several se- mantic tasks, including the identification of semantic relat- edness and similarity. The former task is concerned with whether two words are related or not, independently from their paradigmatic similarity. The identification of seman- tic similarity, instead, is a more specific task and it consists in identifying words that are paradigmatically related (Sun, 2014; Murphy, 2003). DSMs have successfully addressed these two tasks by using vector cosine as a measure of sim- ilarity (Agirre et al., 2009). Unfortunately, however, DSMs are not yet able to discrimi- nate the several semantic relations that exist between words (Santus et al., 2014c). For example, a word such as ⿓ (long, dragon) can be in relation with several other words: 1. Antonymy: ⿓ (long, dragon) vs. 鳳 1 (feng, phoenix) 2. Hypernymy: ⿓ (long, dragon) vs. ⽔ 中 ⽣ 物 (shuizhongshengwu, aquatic life) 3. Nearsynonymy 2: ⿓ (long, dragon) vs 兔 (tu, rabbit) 1In Chinese culture, dragon …

Content truncated to 500 words