Attentive Convolution: Equipping CNNs with RNN-style Attention Mechanisms

Attentive Convolution: Equipping CNNs with RNN-style Attention Mechanisms Wenpeng Yin Department of Computer and Information Science, University of Pennsylvania wenpeng@seas.upenn.edu Hinrich Schütze Center for Information and Language Processing, LMU Munich, Germany inquiries@cislmu.org Abstract In NLP, convolutional neural networks (CNNs) have benefited less than recur- rent neural networks (RNNs) from attention mechanisms. We hypothesize that this is be- cause the attention in CNNs has been mainly implemented as attentive pooling (i.e., it is applied to pooling) rather than as attentive convolution (i.e., it is integrated into con- volution). Convolution is the differentiator of CNNs in that it can powerfully model the higher-level representation of a word by taking into account its local fixed-size con- text in the input text tx. In this work, we propose an attentive convolution network, ATTCONV. It extends the context scope of the convolution operation, deriving higher- level features for a word not only from local context, but also from information ex- tracted from nonlocal context by the atten- tion mechanism commonly used in RNNs. This nonlocal context can come (i) from parts of the input text tx that are distant or (ii) from extra (i.e., external) contexts ty. Experiments on sentence modeling with zero-context (sentiment analysis), single- context (textual entailment) and multiple- context (claim verification) demonstrate the effectiveness of ATTCONV in sentence rep- resentation learning with the incorporation of context. In particular, attentive convo- lution outperforms attentive pooling and is a strong competitor to popular attentive RNNs.1 1 Introduction Natural language processing (NLP) has benefited greatly from the resurgence of deep neural net- works (DNNs), thanks to their high performance with less need of engineered features. A DNN typ- ically is composed of a stack of non-linear trans- 1https://github.com/yinwenpeng/Attentive_ Convolution. formation layers, each generating a hidden rep- resentation for the input by projecting the output of a preceding layer into a new space. To date, building a single and static representation to ex- press an input across diverse problems is far from satisfactory. Instead, it is preferable that the rep- resentation of the input vary in different applica- tion scenarios. In response, attention mechanisms (Graves, 2013; Graves et al., 2014) have been pro- posed to dynamically focus on parts of the in- put that are expected to be more specific to the problem. They are mostly implemented based on fine-grained alignments between two pieces of ob- jects, each emitting a dynamic soft-selection to the components of the other, so that the selected ele- ments dominate in the output hidden representa- tion. Attention-based DNNs have demonstrated good performance on many tasks. Convolutional neural networks (CNNs; LeCun et al., 1998) and recurrent neural networks (RNNs; Elman, 1990) are two important types of DNNs. Most work on attention has been done for RNNs. Attention-based RNNs typically take three types of inputs to make a decision at the current step: (i) the current input state, (ii) a representation of local context (computed unidirectionally or bidi- rectionally; Rocktäschel et al. [2016]), and (iii) the attention-weighted sum of hidden states cor- responding …

Content truncated to 500 words