The Forest Convolutional Network: Compositional Distributional Semantics with a Neural Chart and without Binarization

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1155–1164, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. The Forest Convolutional Network: Compositional Distributional Semantics with a Neural Chart and without Binarization Phong Le and Willem Zuidema Institute for Logic, Language and Computation University of Amsterdam, the Netherlands {p.le,zuidema}@uva.nl Abstract According to the principle of composi- tionality, the meaning of a sentence is computed from the meaning of its parts and the way they are syntactically com- bined. In practice, however, the syntactic structure is computed by automatic parsers which are far-from-perfect and not tuned to the specifics of the task. Current re- cursive neural network (RNN) approaches for computing sentence meaning therefore run into a number of practical difficulties, including the need to carefully select a parser appropriate for the task, deciding how and to what extent syntactic context modifies the semantic composition func- tion, as well as on how to transform parse trees to conform to the branching settings (typically, binary branching) of the RNN. This paper introduces a new model, the Forest Convolutional Network, that avoids all of these challenges, by taking a parse forest as input, rather than a single tree, and by allowing arbitrary branching fac- tors. We report improvements over the state-of-the-art in sentiment analysis and question classification. 1 Introduction For many natural language processing tasks we need to compute meaning representations for sen- tences from meaning representations of words. In a recent line of research on ‘recursive neural net- works’ (e.g., Socher et al. (2010)), both the word and sentence representations are vectors, and the word vectors (“embeddings”) are borrowed from work in distributional semantics or neural lan- guage modelling. Sentence representations, in this approach, are computed by recursively applying a neural network that combines two vectors into one (typically according to the syntactic structure pro- vided by an external parser). The network, which thus implements a ‘composition function’, is opti- mized for delivering sentence representations that support a given semantic task: sentiment analysis (Irsoy and Cardie, 2014; Le and Zuidema, 2015), paraphrase detection (Socher et al., 2011), seman- tic relatedness (Tai et al., 2015) etc. Studies with recursive neural networks have yielded promising results on a variety of such tasks. In this paper, we represent a new recursive neu- ral network architecture that fits squarely in this tradition, but aims to solve a number of difficulties that have arisen in existing work. In particular, the model we propose addresses three issues: 1. how to make the composition functions adap- tive, in the sense that they operate adequately for the many different types of combina- tions (e.g., adjective-noun combinations are of a very different type than VP-PP combina- tions); 2. how to deal with different branching factors of nodes in the relevant syntactic trees (i.e., we want to avoid having to binarize syntac- tic trees,1 but also do not want ternary pro- ductions to be completely independent from binary productions); 3. how to deal with uncertainty about …

Content truncated to 500 words