Direct vs. indirect evaluation of distributional thesauri

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1837–1848, Osaka, Japan, December 11-17 2016. Direct vs. indirect evaluation of distributional thesauri Vincent Claveau IRISA - CNRS Campus de Beaulieu 35042 Rennes, France vincent.claveau@irisa.fr Ewa Kijak IRISA - Univ. of Rennes 1 Campus de Beaulieu 35042 Rennes, France ewa.kijak@irisa.fr Abstract With the success of word embedding methods in various Natural Language Processing tasks, all the fields of distributional semantics have experienced a renewed interest. Beside the famous word2vec, recent studies have presented efficient techniques to build distributional thesaurus; in particular, Claveau et al. (2014) have already shown that Information Retrieval (IR) tools and concepts can be successfully used to build a thesaurus. In this paper, we address the problem of the evaluation of such thesauri or embedding models. Several evaluation scenarii are con- sidered: direct evaluation through reference lexicons and specially crafted datasets, and indirect evaluation through a third party tasks, namely lexical subsitution and Information Retrieval. For this latter task, we adopt the query expansion framework proposed by Claveau and Kijak (2016). Through several experiments, we first show that the recent techniques for building distributional thesaurus outperform the word2vec approach, whatever the evaluation scenario. We also high- light the differences between the evaluation scenarii, which may lead to very different conclu- sions when comparing distributional models. Last, we study the effect of some parameters of the distributional models on these various evaluation scenarii. 1 Introduction For years, distributional semantic has aimed at building thesauri (or lexicons) automatically from text corpora. For a given input (ie. a given word), these thesauri identify semantically similar words based on the assumption that they share a distribution similar to the input word’s one. In practice, this distribu- tional assumption is set such that two words would be considered close if their occurrences share similar contexts. These contexts are typically co-occurring words in a limited window around the considered words, or words syntactically linked. Recently, many studies have explored new techniques to repre- sent the word (or phrase or document) through embeddings: most often, words are thus represented in a vector space, such that two words with close meanings are close in this space. One of the most popular approach is the famous word2vec technique (Mikolov et al., 2013). Of course, such embedding tech- niques rely on the same assumption than ”traditional” distributional semantics (although many studies do not acknowledge it clearly). Evaluating these thesauri or embeddings remains a crucial point to assess the quality of the construc- tion methods and parameters used. In this article, we propose to examine this evaluation problem with different evaluation protocols. Indeed, a commonly used approach is to compare the generated thesauri to one or several reference lexicons. This evaluation procedure, called ’intrinsic’, has the advantage of being straightforward and simple as it aims at estimating the quality and completeness of the gener- ated thesaurus. However, it is based on reference lexicons whose own completeness, quality, or simply their availability for the considered …

Content truncated to 500 words