Do Supervised Distributional Methods Really Learn Lexical Inference Relations?

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 970–976, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Do Supervised Distributional Methods Really Learn Lexical Inference Relations? Omer Levy† Steffen Remus§ Chris Biemann§ Ido Dagan† † Natural Language Processing Lab § Language Technology Lab Bar-Ilan University Technische Universität Darmstadt Ramat-Gan, Israel Darmstadt, Germany {omerlevy,dagan}@cs.biu.ac.il {remus,biem}@cs.tu-darmstadt.de Abstract Distributional representations of words have been recently used in supervised settings for recognizing lexical inference relations be- tween word pairs, such as hypernymy and en- tailment. We investigate a collection of these state-of-the-art methods, and show that they do not actually learn a relation between two words. Instead, they learn an independent property of a single word in the pair: whether that word is a “prototypical hypernym”. 1 Introduction Inference in language involves recognizing infer- ence relations between two words (x and y), such as causality (flu → fever), hypernymy (cat → animal), and other notions of lexical entailment. The distributional approach to automatically recog- nize these relations relies on representing each word x as a vector ⃗x of contextual features: other words that tend to appear in its vicinity. Such features are typically used in word similarity tasks, where cosine similarity is a standard similarity measure between two word vectors: sim(x, y) = cos(⃗x, ⃗y). Many unsupervised distributional methods of rec- ognizing lexical inference replace cosine similarity with an asymmetric similarity function (Weeds and Weir, 2003; Clarke, 2009; Kotlerman et al., 2010; Santus et al., 2014). Supervised methods, reported to perform better, try to learn the asymmetric opera- tor from a training set. The various supervised meth- ods differ by the way they represent each candidate pair of words (x, y): Baroni et al. (2012) use con- catenation ⃗x ⊕ ⃗y, others (Roller et al., 2014; Weeds et al., 2014; Fu et al., 2014) take the vectors’ differ- ence ⃗y − ⃗x, and more sophisticated representations, based on contextual features, have also been tested (Turney and Mohammad, 2014; Rimell, 2014). In this paper, we argue that these supervised meth- ods do not, in fact, learn to recognize lexical infer- ence. Our experiments reveal that much of their pre- viously perceived success stems from lexical mem- orizing. Further experiments show that these super- vised methods learn whether y is a “prototypical hy- pernym” (i.e. a category), regardless of x, rather than learning a concrete relation between x and y. Our mathematical analysis reveals that said meth- ods ignore the interaction between x and y, explain- ing our empirical findings. We modify them ac- cordingly by incorporating the similarity between x and y. Unfortunately, the improvement in per- formance is incremental. We suspect that methods based solely on contextual features of single words are not learning lexical inference relations because contextual features might lack the necessary infor- mation to deduce how one word relates to another. 2 Experiment Setup Due to various differences (e.g. corpora, train/test splits), we do not list previously reported …

Content truncated to 500 words