Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 971–981, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model Nghia The Pham Germ´an Kruszewski Angeliki Lazaridou Marco Baroni Center for Mind/Brain Sciences University of Trento {thenghia.pham|german.kruszewski|angeliki.lazaridou|marco.baroni}@unitn.it Abstract We introduce C-PHRASE, a distributional semantic model that learns word repre- sentations by optimizing context predic- tion for phrases at all levels in a syntactic tree, from single words to full sentences. C-PHRASE outperforms the state-of-the- art C-BOW model on a variety of lexical tasks. Moreover, since C-PHRASE word vectors are induced through a composi- tional learning objective (modeling the contexts of words combined into phrases), when they are summed, they produce sen- tence representations that rival those gen- erated by ad-hoc compositional models. 1 Introduction Distributional semantic models, that induce vector-based meaning representations from pat- terns of co-occurrence of words in corpora, have proven very successful at modeling many lexical relations, such as synonymy, co-hyponomy and analogy (Mikolov et al., 2013c; Turney and Pan- tel, 2010). The recent evaluation of Baroni et al. (2014b) suggests that the C-BOW model intro- duced by Mikolov et al. (2013a) is, consistently, the best across many tasks.1 Interestingly, C-BOW vectors are estimated with a simple compositional approach: The weights of adjacent words are jointly optimized so that their sum will predict the distribution of their contexts. This is reminiscent of how the parame- ters of some compositional distributional seman- 1We refer here not only to the results reported in Baroni et al. (2014b), but also to the more exten- sive evaluation that Baroni and colleagues present in the companion website (http://clic.cimec.unitn. it/composes/semantic-vectors.html). The ex- periments there suggest that only the Glove vectors of Pen- nington et al. (2014) are competitive with C-BOW, and only when trained on a corpus several orders of magnitude larger than the one used for C-BOW. tic models are estimated by optimizing the pre- diction of the contexts in which phrases occur in corpora (Baroni and Zamparelli, 2010; Guevara, 2010; Dinu et al., 2013). However, these compo- sitional approaches assume that word vectors have already been constructed, and contextual evidence is only used to induce optimal combination rules to derive representations of phrases and sentences. In this paper, we follow through on this observa- tion to propose the new C-PHRASE model. Sim- ilarly to C-BOW, C-PHRASE learns word repre- sentations by optimizing their joint context pre- diction. However, unlike in flat, window-based C-BOW, C-PHRASE groups words according to their syntactic structure, and it simultaneously op- timizes context-predictions at different levels of the syntactic hierarchy. For example, given train- ing sentence “A sad dog is howling in the park”, C-PHRASE will optimize context prediction for dog, sad dog, a sad dog, a sad dog is howling, etc., but not, for example, for howling in, as these two words do not form a syntactic constituent by …

Content truncated to 500 words