Word Embedding Evaluation Datasets and Wikipedia Title Embedding for Chinese

Word Embedding Evaluation Datasets and Wikipedia Title Embedding for Chinese Chi-Yen Chen, Wei-Yun Ma Institute of Information Science, Academia Sinica 128 Academia Road, Section 2, Nankang, Taipei, Taiwan 115 {ccy9332, ma}@iis.sinica.edu.tw Abstract Distributed word representations are widely used in many NLP tasks, and there are lots of benchmarks to evaluate word embeddings in English. However there are barely evaluation sets with large enough amount of data for Chinese word embeddings. Therefore, in this paper, we create several evaluation sets for Chinese word embedding on both word similarity task and analogical task via translating some existing popular evaluation sets from English to Chinese. To assess the quality of translated datasets, we obtain human rating from both experts and Amazon Mechanical Turk workers. While translating the datasets, we find out that around 30 percents of word pairs in the benchmarks are Wikipedia titles. This motivate us to evaluate the performance of Wikipedia title embeddings on our new benchmarks. Thus, in this paper, not only the new benchmarks are tested but some new improved approaches of Wikipedia title embeddings are proposed. We perform training of embeddings of Wikipedia titles using not only their Wikipedia context but also their Wikipedia categories, most of categories are noun phrases, and we identify the head words of the noun phrases by a parser for further emphasizing their roles on the training of title embeddings. Experimental results and the comprehensive error analysis demonstrate that the benchmarks can precisely reflect the approaches’ quality, and the effectiveness of our improved approaches on Wikipedia title embeddings are also verified and analyzed in detail. Keywords: word embedding, word embedding benchmark, Wikipedia title embedding, Wikipedia category, knowledge base 1. Introduction Word embeddings are widely used in various natural lan- guage processing (NLP) tasks. Researches evaluates word embeddings on some extrinsic, practical NLP tasks, and also some intrinsic tasks. The two most popular kinds of intrinsic tasks are word similarity and analogical rea- soning, and there are an array of benchmarks for the two kind of tasks in English. Take word similarity for instance, the MEN-3k (Bruni et al., 2012) dataset consists of 3,000 word pairs. The MTurk-287 (Radinsky et al., 2011) dataset consists of 287 word pairs. The SimLex-999 (Hill et al., 2016) consists of 999 word pairs. For analogical reason- ing tasks, the dataset is composed of analogous word pairs. Each word pair is a tuple of word relations that follow a common syntactic relation. For instance, the Google anal- ogy (Mikolov et al., 2013) dataset consists of 19,544 ques- tion pairs. Though there are lots of benchmarks to evaluate word em- beddings in English, there are barely evaluation sets with large enough amount of data for Chinese word embed- dings. For example, researchers at York University released a dataset1 for Chinese word similarity; nevertheless, there are only 50 word pairs in the trial data and around 500 word pairs in the full dataset. Chen and Ma (2017) also created a small amount of datasets for Chinese word embedding, but the quantity …

Content truncated to 500 words