A Convolutional Neural Network for Modelling Sentences

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 655–665, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics A Convolutional Neural Network for Modelling Sentences Nal Kalchbrenner Edward Grefenstette {nal.kalchbrenner, edward.grefenstette, phil.blunsom}@cs.ox.ac.uk Department of Computer Science University of Oxford Phil Blunsom Abstract The ability to accurately represent sen- tences is central to language understand- ing. We describe a convolutional architec- ture dubbed the Dynamic Convolutional Neural Network (DCNN) that we adopt for the semantic modelling of sentences. The network uses Dynamic k-Max Pool- ing, a global pooling operation over lin- ear sequences. The network handles input sentences of varying length and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations. The network does not rely on a parse tree and is easily ap- plicable to any language. We test the DCNN in four experiments: small scale binary and multi-class sentiment predic- tion, six-way question classification and Twitter sentiment prediction by distant su- pervision. The network achieves excellent performance in the first three tasks and a greater than 25% error reduction in the last task with respect to the strongest baseline. 1 Introduction The aim of a sentence model is to analyse and represent the semantic content of a sentence for purposes of classification or generation. The sen- tence modelling problem is at the core of many tasks involving a degree of natural language com- prehension. These tasks include sentiment analy- sis, paraphrase detection, entailment recognition, summarisation, discourse analysis, machine trans- lation, grounded language learning and image re- trieval. Since individual sentences are rarely ob- served or not observed at all, one must represent a sentence in terms of features that depend on the words and short n-grams in the sentence that are frequently observed. The core of a sentence model involves a feature function that defines the process The cat sat on the red mat The cat sat on the red mat Figure 1: Subgraph of a feature graph induced over an input sentence in a Dynamic Convolu- tional Neural Network. The full induced graph has multiple subgraphs of this kind with a distinct set of edges; subgraphs may merge at different layers. The left diagram emphasises the pooled nodes. The width of the convolutional filters is 3 and 2 respectively. With dynamic pooling, a fil- ter with small width at the higher layers can relate phrases far apart in the input sentence. by which the features of the sentence are extracted from the features of the words or n-grams. Various types of models of meaning have been proposed. Composition based methods have been applied to vector representations of word meaning obtained from co-occurrence statistics to obtain vectors for longer phrases. In some cases, com- position is defined by algebraic operations over word meaning vectors to produce sentence mean- ing vectors (Erk and Pad´o, 2008; Mitchell and Lapata, 2008; Mitchell and Lapata, 2010; Tur- ney, 2012; Erk, 2012; Clarke, 2012). In other cases, a composition …

Content truncated to 500 words