Deriving Adjectival Scales from Continuous Space Word Representations

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1625–1630, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Deriving adjectival scales from continuous space word representations Joo-Kyung Kim Department of Computer Science and Engineering The Ohio State University Columbus, OH 43210, USA kimjook@cse.ohio-state.edu Marie-Catherine de Marneffe Department of Linguistics The Ohio State University Columbus, OH 43210, USA mcdm@ling.ohio-state.edu Abstract Continuous space word representations ex- tracted from neural network language mod- els have been used effectively for natural lan- guage processing, but until recently it was not clear whether the spatial relationships of such representations were interpretable. Mikolov et al. (2013) show that these representations do capture syntactic and semantic regularities. Here, we push the interpretation of continuous space word representations further by demon- strating that vector offsets can be used to de- rive adjectival scales (e.g., okay < good < ex- cellent). We evaluate the scales on the indirect answers to yes/no questions corpus (de Marn- effe et al., 2010). We obtain 72.8% accuracy, which outperforms previous results (∼60%) on this corpus and highlights the quality of the scales extracted, providing further support that the continuous space word representations are meaningful. 1 Introduction There has recently been a surge of interest for deep learning in natural language processing. In particu- lar, neural network language models (NNLMs) have been used to learn distributional word vectors (Ben- gio et al., 2003; Schwenk, 2007; Mikolov et al., 2010): the models jointly learn an embedding of words into an n-dimensional feature space. One of the advantages put forth for such distributed rep- resentations compared to traditional n-gram mod- els is that similar words are likely to have similar vector representations in a continuous space model, whereas the discrete units of an n-gram model do not exhibit any inherent relation with one another. It has been shown that the continuous space repre- sentations improve performance in a variety of NLP tasks, such as POS tagging, semantic role labeling, named entity resolution, parsing (Collobert and We- ston, 2008; Turian et al., 2010; Huang et al., 2012). Mikolov et al. (2013) show that there are some syntactic and semantic regularities in the word rep- resentations learned, such as the singular/plural rela- tion (the difference of singular and plural word vec- tors are equivalent: apple − apples ≈ car − cars ≈ family − families) or the gender relation (a mascu- line noun can be transformed into the feminine form: king − man + woman ≈ queen). We extend Mikolov et al. (2013)’s approach and explore further the interpretation of the vector space. We show that the word vectors learned by NNLMs are meaningful: we can extract scalar relationships between adjectives (e.g., bad < okay < good < ex- cellent), which can not only serve to build a senti- ment lexicon but also be used for inference. To eval- uate the quality of the scalar relationships learned by NNLMs, we use the indirect yes/no question an- swer pairs (IQAP) from (de …

Content truncated to 500 words