Symmetric Patterns and Coordinations: Fast and Enhanced Representations of Verbs and Adjectives

Proceedings of NAACL-HLT 2016, pages 499–505, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Symmetric Patterns and Coordinations: Fast and Enhanced Representations of Verbs and Adjectives Roy Schwartz,1 Roi Reichart,2 1Institute of Computer Science, The Hebrew University 2Faculty of Industrial Engineering and Management, Technion, IIT {roys02|arir}@cs.huji.ac.il roiri@ie.technion.ac.il Ari Rappoport1 Abstract State-of-the-art word embeddings, which are often trained on bag-of-words (BOW) con- texts, provide a high quality representation of aspects of the semantics of nouns. However, their quality decreases substantially for the task of verb similarity prediction. In this paper we show that using symmetric pattern contexts (SPs, e.g., “X and Y”) improves word2vec verb similarity performance by up to 15% and is also instrumental in adjective similarity pre- diction. The unsupervised SP contexts are even superior to a variety of dependency con- texts extracted using a supervised dependency parser. Moreover, we observe that SPs and dependency coordination contexts (Coor) cap- ture a similar type of information, and demon- strate that Coor contexts are superior to other dependency contexts including the set of all dependency contexts, although they are still inferior to SPs. Finally, there are substantially fewer SP contexts compared to alternative rep- resentations, leading to a massive reduction in training time. On an 8G words corpus and a 32 core machine, the SP model trains in 11 min- utes, compared to 5 and 11 hours with BOW and all dependency contexts, respectively. 1 Introduction In recent years, vector space models (VSMs) have become prominent in NLP. VSMs are often eval- uated by measuring their ability to predict human judgments of lexical semantic relations between pairs of words, mostly association or similarity. While many datasets for these tasks are limited to pairs of nouns, the recent SimLex999 word similar- ity dataset (Hill et al., 2014) also consists of sim- ilarity scores for verb and adjective pairs. State-of- the-art VSMs such as word2vec skip-gram (w2v-SG, (Mikolov et al., 2013a)) and GloVe (Pennington et al., 2014) excel at noun-related tasks. However, their performance substantially decreases on verb similar- ity prediction in SimLex999, and their adjective rep- resentations have rarely been evaluated (Section 2). In this paper we show that a key factor in the re- duced performance of the w2v-SG model on verb representation is its reliance on bag-of-words (BOW) contexts: contexts of the represented words that con- sist of words in their physical proximity. We investi- gate a number of alternative contexts for this model, including various dependency contexts, and show that simple, automatically acquired symmetric pat- terns (SPs, e.g., “X or Y”, (Hearst, 1992; Davidov and Rappoport, 2006)) are the most useful contexts for the representation of verbs and also adjectives. Moreover, the SP-based model is much more com- pact than the alternatives, making its training an or- der of magnitude faster. In particular, we train several versions of the w2v- SG model, each with a different context type, and evaluate the resulting word embeddings on the task of predicting the similarity scores of the verb and …

Content truncated to 500 words