Adaptive Joint Learning of Compositional and Non-Compositional Phrase Embeddings

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 205–215, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Adaptive Joint Learning of Compositional and Non-Compositional Phrase Embeddings Kazuma Hashimoto and Yoshimasa Tsuruoka The University of Tokyo, 3-7-1 Hongo, Bunkyo-ku, Tokyo, Japan {hassy,tsuruoka}@logos.t.u-tokyo.ac.jp Abstract We present a novel method for jointly learning compositional and non- compositional phrase embeddings by adaptively weighting both types of em- beddings using a compositionality scoring function. The scoring function is used to quantify the level of compositionality of each phrase, and the parameters of the function are jointly optimized with the ob- jective for learning phrase embeddings. In experiments, we apply the adaptive joint learning method to the task of learning embeddings of transitive verb phrases, and show that the compositionality scores have strong correlation with human ratings for verb-object compositionality, substantially outperforming the previous state of the art. Moreover, our embeddings improve upon the previous best model on a transitive verb disambiguation task. We also show that a simple ensemble technique further improves the results for both tasks. 1 Introduction Representing words and phrases in a vector space has proven effective in a variety of language pro- cessing tasks (Pham et al., 2015; Sutskever et al., 2014). In most of the previous work, phrase em- beddings are computed from word embeddings by using various kinds of composition functions. Such composed embeddings are called composi- tional embeddings. An alternative way of comput- ing phrase embeddings is to treat phrases as single units and assigning a unique embedding to each candidate phrase (Mikolov et al., 2013; Yazdani et al., 2015). Such embeddings are called non- compositional embeddings. Relying solely on non-compositional embed- dings has the obvious problem of data sparsity (i.e. rare or unknown phrase problems). At the same time, however, using compositional embeddings is not always the best option since some phrases are inherently non-compositional. For example, the phrase “bear fruits” means “to yield results”1 but it is hard to infer its meaning by composing the meanings of “bear” and “fruit”. Treating all phrases as compositional also has a negative ef- fect in learning the composition function because the words in those idiomatic phrases are not just uninformative but can serve as noisy samples in the training. These problems have motivated us to adaptively combine both types of embeddings. Most of the existing methods for learning phrase embeddings can be divided into two ap- proaches. One approach is to learn compositional embeddings by regarding all phrases as composi- tional (Pham et al., 2015; Socher et al., 2012). The other approach is to learn both types of embed- dings separately and use the better ones (Kartsak- lis et al., 2014; Muraoka et al., 2014). Kartsaklis et al. (2014) show that non-compositional embed- dings are better suited for a phrase similarity task, whereas Muraoka et al. (2014) report the opposite results on other tasks. These results suggest that we should not stick to either of the two types of embeddings unconditionally and …

Content truncated to 500 words