Jointly Learning Word Representations and Composition Functions Using Predicate-Argument Structures

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1544–1555, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Jointly Learning Word Representations and Composition Functions Using Predicate-Argument Structures Kazuma Hashimoto†, Pontus Stenetorp†, Makoto Miwa‡, and Yoshimasa Tsuruoka† †The University of Tokyo, 3-7-1 Hongo, Bunkyo-ku, Tokyo, Japan {hassy,pontus,tsuruoka}@logos.t.u-tokyo.ac.jp ‡Toyota Technological Institute, 2-12-1 Hisakata, Tempaku-ku, Nagoya, Japan makoto-miwa@toyota-ti.ac.jp Abstract We introduce a novel compositional lan- guage model that works on Predicate- Argument Structures (PASs). Our model jointly learns word representations and their composition functions using bag- of-words and dependency-based con- texts. Unlike previous word-sequence- based models, our PAS-based model com- poses arguments into predicates by using the category information from the PAS. This enables our model to capture long- range dependencies between words and to better handle constructs such as verb- object and subject-verb-object relations. We verify this experimentally using two phrase similarity datasets and achieve re- sults comparable to or higher than the pre- vious best results. Our system achieves these results without the need for pre- trained word vectors and using a much smaller training corpus; despite this, for the subject-verb-object dataset our model improves upon the state of the art by as much as ∼10% in relative performance. 1 Introduction Studies on embedding single words in a vector space have made notable successes in capturing their syntactic and semantic properties (Turney and Pantel, 2010). These embeddings have also been found to be a useful component for Natural Language Processing (NLP) systems; for exam- ple, Turian et al. (2010) and Collobert et al. (2011) demonstrated how low-dimensional word vectors learned by Neural Network Language Models (NNLMs) are beneficial for a wide range of NLP tasks. Recently, the main focus of research on vector space representation is shifting from word repre- sentations to phrase representations (Baroni and Zamparelli, 2010; Grefenstette and Sadrzadeh, 2011; Mitchell and Lapata, 2010; Socher et al., 2012). Combining the ideas of NNLMs and se- mantic composition, Tsubaki et al. (2013) intro- duced a novel NNLM incorporating verb-object dependencies. More recently, Levy and Goldberg (2014) presented a NNLM that integrated syntac- tic dependencies. However, to the best of our knowledge, there is no previous work on integrat- ing a variety of syntactic and semantic dependen- cies into NNLMs in order to learn composition functions as well as word representations. The fol- lowing question thus arises naturally: Can a variety of dependencies be used to jointly learn both stand-alone word vectors and their compositions, embedding them in the same vector space? In this work, we bridge the gap between purely context-based (Levy and Goldberg, 2014; Mikolov et al., 2013b; Mnih and Kavukcuoglu, 2013) and compositional (Tsubaki et al., 2013) NNLMs by using the flexible set of categories from Predicate-Argument-Structures (PASs). More specifically, we propose a Compositional Log-Bilinear Language Model using PASs (PAS- CLBLM), an overview of which is shown in Figure 1. The model is trained by maximizing the accuracy of predicting target words from their bag-of-words and dependency-based context, which provides …

Content truncated to 500 words