Finding The Best Model Among Representative Compositional Models

PACLIC 28 !65 Finding The Best Model Among Representative Compositional Models Masayasu Muraoka† Sonse Shimaoka‡ Kazeto Yamamoto† Yotaro Watanabe† Naoaki Okazaki†∗ Kentaro Inui† Tohoku University†‡ Japan Science and Technology Agency (JST)∗ {muraoka,kazeto,yotaro-w,okazaki,inui} @ecei.tohoku.ac.jp† simaokasonse@yahoo.co.jp‡ Abstract The field of distributional-compositional se- mantics has yielded a range of computational models for composing the vector of a phrase from those of constituent word vectors. Ex- isting models have various ranges of their expressiveness, recursivity, and trainability. However, these models have not been exam- ined closely for their compositionality. We implement and compare these models under the same conditions. The experimentally ob- tained results demonstrate that the model us- ing different composition matrices for differ- ent dependency relations achieved state-of- the-art performance on a dataset for two-word compositions (Mitchell and Lapata, 2010). 1 Introduction Computing the meaning of a text has posed a chal- lenge in NLP for many years. Based on the distri- butional hypothesis (Firth, 1957), the meaning of a word is typically represented as a real-valued vector, with elements representing the frequencies of words that co-occur in the context of the word in a cor- pus. Numerous studies have demonstrated learned word vectors from a large text corpus (Bullinaria and Levy, 2007; Collobert and Weston, 2008; Tur- ney and Pantel, 2010; Mnih and Kavukcuoglu, 2013; Mikolov et al., 2013). In contrast, the same approach is not scalable to a complex linguistic unit (e.g., phrase or sentence) because of the data sparseness problem: the longer the length of a phrase, the fewer times the phrase oc- curs in a corpus. For this reason, we cannot acquire semantic information reliably from co-occurrence statistics of a phrase. Recently, numerous studies have explored compositional semantics, in which the meaning of a phrase, clause, or sentence is com- puted from those of its constituents (Mitchell and Lapata, 2008; Mitchell and Lapata, 2010; Guevara, 2010; Zanzotto et al., 2010; Socher et al., 2011; Ba- roni et al., 2012; Socher et al., 2012; Socher et al., 2013a; Socher et al., 2014). These studies mostly address theories and methods for computing a vec- tor of a phrase from the vectors of its constituents; the simplest but effective approach is to take the av- erage of the two input vectors. A simple approach such as additive and multi- plicative compositions has been a strong baseline over more complex models (Blacoe and Lapata, 2012; Socher et al., 2013b). However, Erk and Pad´o (2008) argued the importance of syntax relations: the simple additive/multiplicative approach yields the same vector for phrases a horse draws and draw a horse, ignoring the syntactic structure by which horse in the former phrase is a subject whereas horse in the latter is the object. They formulated a gener- alized composition function including such a com- position. However, this generalized composition is too complex to learn. These models usually do not work well for now. As described in this paper, through a human- correlation experiment, we explore the most useful model among the representative models that have been …

Content truncated to 500 words