Automatic Selection of Context Configurations for Improved Class-Specific Word Representations

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 112–122, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics Automatic Selection of Context Configurations for Improved Class-Specific Word Representations Ivan Vuli´c1, Roy Schwartz2,3, Ari Rappoport4 Roi Reichart5, Anna Korhonen1 1 Language Technology Lab, DTAL, University of Cambridge 2 CS & Engineering, University of Washington and 3Allen Institute for AI 4 Institute of Computer Science, The Hebrew University of Jerusalem 5 Faculty of Industrial Engineering and Management, Technion, IIT {iv250,alk23}@cam.ac.uk roysch@cs.washington.edu arir@cs.huji.ac.il roiri@ie.technion.ac.il Abstract This paper is concerned with identifying contexts useful for training word represen- tation models for different word classes such as adjectives (A), verbs (V), and nouns (N). We introduce a simple yet ef- fective framework for an automatic selec- tion of class-specific context configurations. We construct a context configuration space based on universal dependency relations between words, and efficiently search this space with an adapted beam search algo- rithm. In word similarity tasks for each word class, we show that our framework is both effective and efficient. Particularly, it improves the Spearman’s ρ correlation with human scores on SimLex-999 over the best previously proposed class-specific contexts by 6 (A), 6 (V) and 5 (N) ρ points. With our selected context configurations, we train on only 14% (A), 26.2% (V), and 33.6% (N) of all dependency-based contexts, resulting in a reduced training time. Our results gen- eralise: we show that the configurations our algorithm learns for one English training setup outperform previously proposed con- text types in another training setup for En- glish. Moreover, basing the configuration space on universal dependencies, it is possi- ble to transfer the learned configurations to German and Italian. We also demonstrate improved per-class results over other con- text types in these two languages. 1 Introduction Dense real-valued word representations (embed- dings) have become ubiquitous in NLP, serving as invaluable features in a broad range of tasks (Turian et al., 2010; Collobert et al., 2011; Chen and Manning, 2014). The omnipresent word2vec skip-gram model with negative sampling (SGNS) (Mikolov et al., 2013) is still considered a ro- bust and effective choice for a word representation model, due to its simplicity, fast training, as well as its solid performance across semantic tasks (Baroni et al., 2014; Levy et al., 2015). The original SGNS implementation learns word representations from local bag-of-words contexts (BOW). However, the underlying model is equally applicable with other context types (Levy and Goldberg, 2014a). Recent work suggests that “not all contexts are created equal”. For example, reaching beyond stan- dard BOW contexts towards contexts based on de- pendency parses (Bansal et al., 2014; Melamud et al., 2016) or symmetric patterns (Schwartz et al., 2015, 2016) yields significant improvements in learning representations for particular word classes such as adjectives (A) and verbs (V). Moreover, Schwartz et al. (2016) demonstrated that a subset of dependency-based contexts which covers only coordination structures is particularly effective for SGNS training, both in terms of the quality of the …

Content truncated to 500 words