Sense and Similarity: A Study of Sense-level Similarity Measures

Proceedings of the Third Joint Conference on Lexical and Computational Semantics (*SEM 2014), pages 30–39, Dublin, Ireland, August 23-24 2014. Sense and Similarity: A Study of Sense-level Similarity Measures Nicolai Erbs†, Iryna Gurevych†‡ and Torsten Zesch§ † UKP Lab, Technische Universit¨at Darmstadt ‡ Information Center for Education, DIPF, Frankfurt § Language Technology Lab, University of Duisburg-Essen http://www.ukp.tu-darmstadt.de Abstract In this paper, we investigate the differ- ence between word and sense similarity measures and present means to convert a state-of-the-art word similarity measure into a sense similarity measure. In or- der to evaluate the new measure, we cre- ate a special sense similarity dataset and re-rate an existing word similarity dataset using two different sense inventories from WordNet and Wikipedia. We discover that word-level measures were not able to differentiate between different senses of one word, while sense-level measures actually increase correlation when shift- ing to sense similarities. Sense-level sim- ilarity measures improve when evaluated with a re-rated sense-aware gold standard, while correlation with word-level similar- ity measures decreases. 1 Introduction Measuring similarity between words is a very im- portant task within NLP with applications in tasks such as word sense disambiguation, information retrieval, and question answering. However, most of the existing approaches compute similarity on the word-level instead of the sense-level. Conse- quently, most evaluation datasets have so far been annotated on the word level, which is problem- atic as annotators might not know some infrequent senses and are influenced by the more probable senses. In this paper, we provide evidence that this process heavily influences the annotation process. For example, when people are presented the word pair jaguar - gamepad only few people know that Jaguar Gamepad Zoo .0070 .0016 .0000 Figure 1: Similarity between words. jaguar is also the name of an Atari game console.1 People rather know the more common senses of jaguar, i.e. the car brand or the animal. Thus, the word pair receives a low similarity score, while computational measures are not so easily fooled by popular senses. It is thus likely that existing evaluation datasets give a wrong picture of the true performance of similarity measures. Thus, in this paper we investigate whether sim- ilarity should be measured on the sense level. We analyze state-of-the-art methods and describe how the word-based Explicit Semantic Analysis (ESA) measure (Gabrilovich and Markovitch, 2007) can be transformed into a sense-level measure. We create a sense similarity dataset, where senses are clearly defined and evaluate similarity measures with this novel dataset. We also re-annotate an ex- isting word-level dataset on the sense level in order to study the impact of sense-level computation of similarity. 2 Word-level vs. Sense-level Similarity Existing measures either compute similarity (i) on the word level or (ii) on the sense level. Similarity on the word level may cover any possible sense of the word, where on the sense level only the actual sense is considered. We use Wikipedia Link Mea- 1If you knew that it is a certain sign that you are getting old. 30 …

Content truncated to 500 words