Modeling and Learning Semantic Co-Compositionality through Prototype Projections and Neural Networks

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 130–140, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Modeling and Learning Semantic Co-Compositionality through Prototype Projections and Neural Networks Masashi Tsubaki, Kevin Duh, Masashi Shimbo, Yuji Matsumoto Graduate School of Information Science Nara Institute of Science and Technology 8916-5, Takayama, Ikoma, Nara 630-0192, Japan {masashi-t,kevinduh,shimbo,matsu}@is.naist.jp Abstract We present a novel vector space model for se- mantic co-compositionality. Inspired by Gen- erative Lexicon Theory (Pustejovsky, 1995), our goal is a compositional model where both predicate and argument are allowed to modify each others’ meaning representations while generating the overall semantics. This readily addresses some major challenges with current vector space models, notably the pol- ysemy issue and the use of one represen- tation per word type. We implement co- compositionality using prototype projections on predicates/arguments and show that this is effective in adapting their word represen- tations. We further cast the model as a neural network and propose an unsupervised algorithm to jointly train word representations with co-compositionality. The model achieves the best result to date (ρ = 0.47) on the semantic similarity task of transitive verbs (Grefenstette and Sadrzadeh, 2011). 1 Introduction Vector space models of words have been very successful in capturing the semantic and syntactic characteristics of individual lexical items (Turney and Pantel, 2010). Much research has addressed the question of how to construct individual word representations, for example distributional models (Mitchell and Lapata, 2010) and neural models (Collobert and Weston, 2008). These word repre- sentations are used in various natural language pro- cessing (NLP) tasks such as part-of-speech tagging, chunking, named entity recognition, and semantic Vk Pcompany=Vk TVk Co-Compositionality with Prototype Projections Co-compositional vector of verb and object start build buy operate = company run VERB runcompany Prototype Projection companyrun OBJ ・・・ ・・・ Ok firm bank hotel ・・・ ・・・ Prun=Ok TOk ≈ ≈ m × × Figure 1: Here, we capture the semantics of run in run company by projecting the original word representation of run to the prototype space of company (and vice versa). role labeling (Turian et al., 2010; Collobert et al., 2011). Recently, modeling of semantic compositionality (Frege, 1892) in vector space has emerged as another important line of research (Mitchell and Lapata, 2008; Mitchell and Lapata, 2010; Baroni and Zam- parelli, 2010; Socher et al., 2012; Grefenstette and Sadrzadeh, 2011; Van de Cruys et al., 2013). The goal is to formulate how individual word represen- tations ought to be combined to achieve phrasal or sentential semantics. The main questions for semantic compositionality that we are concerned with are: (1) how can poly- semy be handled by a single vector representation per word type, learned by either a distributional or neural model, and (2) how does composition resolve 130 these ambiguities. To this end, we are inspired by the idea of type coercion and co-compositionality in Generative Lexicon Theory (Pustejovsky, 1995). Co-compositionality advocates that instead of a predicate-argument view of composition, both pred- icate and argument influence/coerce …

Content truncated to 500 words