Counter-fitting Word Vectors to Linguistic Constraints

Proceedings of NAACL-HLT 2016, pages 142–148, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Counter-fitting Word Vectors to Linguistic Constraints Nikola Mrkˇsi´c1, Diarmuid ´O S´eaghdha2, Blaise Thomson2, Milica Gaˇsi´c1 Lina Rojas-Barahona1, Pei-Hao Su1, David Vandyke1, Tsung-Hsien Wen1, Steve Young1 1 Department of Engineering, University of Cambridge, UK 2 Apple Inc. {nm480,mg436,phs26,djv27,thw28,sjy}@cam.ac.uk {doseaghdha, blaisethom}@apple.com Abstract In this work, we present a novel counter-fitting method which injects antonymy and synonymy constraints into vector space representations in order to improve the vectors’ capability for judging semantic similarity. Applying this method to publicly available pre-trained word vectors leads to a new state of the art perfor- mance on the SimLex-999 dataset. We also show how the method can be used to tailor the word vector space for the downstream task of dialogue state tracking, resulting in robust im- provements across different dialogue domains. 1 Introduction Many popular methods that induce representations for words rely on the distributional hypothesis – the assumption that semantically similar or related words appear in similar contexts. This hypothesis sup- ports unsupervised learning of meaningful word rep- resentations from large corpora (Curran, 2003; ´O S´eaghdha and Korhonen, 2014; Mikolov et al., 2013; Pennington et al., 2014). Word vectors trained using these methods have proven useful for many down- stream tasks including machine translation (Zou et al., 2013) and dependency parsing (Bansal et al., 2014). One drawback of learning word embeddings from co-occurrence information in corpora is that it tends to coalesce the notions of semantic similarity and con- ceptual association (Hill et al., 2014b). Furthermore, even methods that can distinguish similarity from association (e.g., based on syntactic co-occurrences) will generally fail to tell synonyms from antonyms (Mohammad et al., 2008). For example, words such east expensive British Before west pricey American north cheaper Australian south costly Britain southeast overpriced European northeast inexpensive England After eastward costly Brits eastern pricy London easterly overpriced BBC - pricey UK - afford Britain Table 1: Nearest neighbours for target words using GloVe vectors before and after counter-fitting as east and west or expensive and inexpensive appear in near-identical contexts, which means that distribu- tional models produce very similar word vectors for such words. Examples of such anomalies in GloVe vectors can be seen in Table 1, where words such as cheaper and inexpensive are deemed similar to (their antonym) expensive. A second drawback is that similarity and antonymy can be application- or domain-specific. In our case, we are interested in exploiting distributional knowl- edge for the dialogue state tracking task (DST). The DST component of a dialogue system is responsi- ble for interpreting users’ utterances and updating the system’s belief state – a probability distribution over all possible states of the dialogue. For exam- ple, a DST for the restaurant domain needs to detect whether the user wants a cheap or expensive restau- rant. Being able to generalise using distributional information while still distinguishing between se- mantically different yet conceptually related words 142 (e.g. cheaper and pricey) is critical for the …

Content truncated to 500 words