Predicting the relevance of distributional semantic similarity with contextual information

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 479–488, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Predicting the relevance of distributional semantic similarity with contextual information Philippe Muller IRIT, Toulouse University Universit´e Paul Sabatier 118 Route de Narbonne 31062 Toulouse Cedex 04 philippe.muller@irit.fr C´ecile Fabre CLLE, Toulouse University Universit´e Toulouse-Le Mirail 5 alles A. Machado 31058 Toulouse Cedex cecile.fabre@univ-tlse2.fr Cl´ementine Adam CLLE, Toulouse University Universit´e Toulouse-Le Mirail 5 alles A. Machado 31058 Toulouse Cedex clementine.adam@univ-tlse2.fr Abstract Using distributional analysis methods to compute semantic proximity links be- tween words has become commonplace in NLP. The resulting relations are often noisy or difficult to interpret in general. This paper focuses on the issues of eval- uating a distributional resource and filter- ing the relations it contains, but instead of considering it in abstracto, we focus on pairs of words in context. In a dis- course, we are interested in knowing if the semantic link between two items is a by- product of textual coherence or is irrele- vant. We first set up a human annotation of semantic links with or without contex- tual information to show the importance of the textual context in evaluating the rele- vance of semantic similarity, and to assess the prevalence of actual semantic relations between word tokens. We then built an ex- periment to automatically predict this rel- evance, evaluated on the reliable reference data set which was the outcome of the first annotation. We show that in-document in- formation greatly improve the prediction made by the similarity level alone. 1 Introduction The goal of the work presented in this paper is to improve distributional thesauri, and to help evalu- ate the content of such resources. A distributional thesaurus is a lexical network that lists semantic neighbours, computed from a corpus and a simi- larity measure between lexical items, which gen- erally captures the similarity of contexts in which the items occur. This way of building a seman- tic network has been very popular since (Grefen- stette, 1994; Lin, 1998), even though the nature of the information it contains is hard to define, and its evaluation is far from obvious. A distributional thesaurus includes a lot of “noise” from a seman- tic point of view, but also lists relevant lexical pairs that escape classical lexical relations such as syn- onymy or hypernymy. There is a classical dichotomy when evaluat- ing NLP components between extrinsic and in- trinsic evaluations (Jones, 1994), and this applies to distributional thesauri (Curran, 2004; Poibeau and Messiant, 2008). Extrinsic evaluations mea- sure the capacity of a system in which a resource or a component to evaluate has been used, for in- stance in this case information retrieval (van der Plas, 2008) or word sense disambiguation (Weeds and Weir, 2005). Intrinsic evaluations try to mea- sure the resource itself with respect to some hu- man standard or judgment, for instance by com- paring a distributional resource with respect to an existing synonym dictionary or similarity …

Content truncated to 500 words