Semantic Relatedness of Wikipedia Concepts – Benchmark Data and a Working Solution

Semantic Relatedness of Wikipedia Concepts – Benchmark Data and a Working Solution Liat Ein Dor, Alon Halfon, Yoav Kantor, Ran Levy, Yosi Mass, Ruty Rinott, Eyal Shnarch, Noam Slonim IBM Haifa Reseacrh Lab Haifa University, Mount Carmel, Haifa, HA 31905, Israel {liate,alonhal,yoavka,ranl,yosimass,rutyr,eyals,noams}@il.ibm.com Abstract Wikipedia is a very popular source of encyclopedic knowledge which provides highly reliable articles in a variety of domains. This richness and popularity created a strong motivation among NLP researchers to develop relatedness measures between Wikipedia concepts. In this paper, we introduce WORD (Wikipedia Oriented Relatedness Dataset), a new type of concept relatedness dataset, composed of 19,276 pairs of Wikipedia concepts. This is the first human annotated dataset of Wikipedia concepts, whose purpose is twofold. On the one hand, it can serve as a benchmark for evaluating concept-relatedness methods. On the other hand, it can be used as supervised data for developing new models for concept relatedness prediction. Among the advantages of this dataset compared to its term-relatedness counterparts, are its built-in disambiguation solution, and its richness with meaningful multi- word terms. Based on this benchmark we develop a new tool, named WORT (Wikipedia Oriented Relatedness Tool), for measuring the level of relatedness between pairs of concepts. We show that the relatedness predictions of WORT outperform state of the art methods. Keywords: Concept Relatedness, Relatedness function, Dataset 1. Introduction Wikipedia is the leading open encyclopedia with very good coverage on diverse topics. Many terms encountered in hu- man written texts are in fact Wikipedia concepts. As a re- sult, multiple NLP tasks, such as entity linking and docu- ment clustering, can benefit from quantitatively measuring the level of relatedness between such concepts. A variety of methods have been proposed for measuring relatedness be- tween Wikipedia concepts, e.g. (Witten and Milne, 2008; Strube and Ponzetto, 2006; Gabrilovich and Markovitch, 2007; Sherkat and Milios, 2017). Evaluating these meth- ods, and developing new ones naturally requires high qual- ity large benchmark data. Here, we introduce a new dataset, named WORD, composed of 19, 276 Wikipedia concept pairs, manually annotated to determine their level of relat- edness. We exploit these data to assess several types of state of the art semantic-relatedness tools, including word and document similarity functions (Mikolov et al., 2013; Gabrilovich and Markovitch, 2007), and link based meth- ods (Witten and Milne, 2008; Ceccarelli et al., 2013), for the task of concept relatedness. Moreover, we sug- gest several new utilities, that are explicitly designed for this task, such as PMI (Church and Hanks, 1990; Bulli- naria and Levy, 2007) measured between concepts based on their statistical co-occurrence in the entire Wikipedia corpus. We further exploit the data for supervised learn- ing of concept-relatedness function, and use our novel utili- ties along with known state-of-the-art semantic-relatedness methods as features in a Linear Regression (LR) model. The resultant concept-relatedness tool, termed henceforth WORT, clearly outperforms each individual feature. Finally, to demonstrate the versatility of WORD, we sug- gest a mechanism for automatically generating a dis- ambiguated term-relatedness dataset from a Wikipedia …

Content truncated to 500 words