The Lifted Matrix-Space Model for Semantic Composition

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 508–518 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 508 The Lifted Matrix-Space Model for Semantic Composition WooJin Chung1 woojin@nyu.edu Sheng-Fu Wang1 shengfu.wang@nyu.edu Samuel R. Bowman1,2 bowman@nyu.edu 1Dept. of Linguistics New York University 10 Washington Place New York, NY 10003 2Center for Data Science New York University 60 Fifth Avenue New York, NY 10011 Abstract Tree-structured neural network architectures for sentence encoding draw inspiration from the approach to semantic composition gener- ally seen in formal linguistics, and have shown empirical improvements over comparable se- quence models by doing so. Moreover, adding multiplicative interaction terms to the com- position functions in these models can yield significant further improvements. However, existing compositional approaches that adopt such a powerful composition function scale poorly, with parameter counts exploding as model dimension or vocabulary size grows. We introduce the Lifted Matrix-Space model, which uses a global transformation to map vector word embeddings to matrices, which can then be composed via an operation based on matrix-matrix multiplication. Its compo- sition function effectively transmits a larger number of activations across layers with rela- tively few model parameters. We evaluate our model on the Stanford NLI corpus, the Multi- Genre NLI corpus, and the Stanford Sentiment Treebank and find that it consistently outper- forms TreeLSTM (Tai et al., 2015), the previ- ous best known composition function for tree- structured models. 1 Introduction Contemporary theoretical accounts of natural lan- guage syntax and semantics consistently hold that sentences are tree-structured, and that the mean- ing of each node in each tree is calculated from the meaning of its child nodes using a relatively sim- ple semantic composition process which is applied recursively bottom-up (Chierchia and McConnell- Ginet, 1990; Dowty, 2007). In tree-structured re- cursive neural networks (TreeRNN; Socher et al., 2010), a similar procedure is used to build repre- sentations for sentences for use in natural language understanding tasks, with distributed representa- tions for words repeatedly fed through a neural Figure 1: The Lifted Matrix-Space model in schematic form. Words are stored as vectors and projected into matrix space by the LIFT layer. A parametric COMPO- SITION function combines pairs of these matrices using multiplicative interactions. network composition function according to a bi- nary tree structure supplied by a parser. The suc- cess of a tree-structured model largely depends on the design of its composition function. It has been repeatedly shown that a composition function that captures multiplicative interactions between the two items being composed yields bet- ter results (Rudolph and Giesbrecht, 2010; Socher et al., 2012, 2013) than do otherwise-equivalent functions based on simple linear interactions. This paper presents a novel model which advances this line of research, the Lifted Matrix-Space model. We utilize a tensor-parameterized LIFT layer that learns to produce matrix representations of words that are dependent on the content of pre-trained word embedding vectors. Composition of two ma- trix representations is carried out by a composition layer, into …

Content truncated to 500 words