Learning Generic Sentence Representations Using Convolutional Neural Networks

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2390–2400 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Learning Generic Sentence Representations Using Convolutional Neural Networks Zhe Gan†, Yunchen Pu†, Ricardo Henao†, Chunyuan Li†, Xiaodong He‡, Lawrence Carin† †Duke University, ‡Microsoft Research, Redmond, WA 98052, USA {zg27, yp42, r.henao, cl319, lcarin}@duke.edu xiaohe@microsoft.com Abstract We propose a new encoder-decoder ap- proach to learn distributed sentence rep- resentations that are applicable to multiple purposes. The model is learned by using a convolutional neural network as an encoder to map an input sentence into a continuous vector, and using a long short-term mem- ory recurrent neural network as a decoder. Several tasks are considered, including sen- tence reconstruction and future sentence prediction. Further, a hierarchical encoder- decoder model is proposed to encode a sen- tence to predict multiple future sentences. By training our models on a large collection of novels, we obtain a highly generic con- volutional sentence encoder that performs well in practice. Experimental results on several benchmark datasets, and across a broad range of applications, demonstrate the superiority of the proposed model over competing methods. 1 Introduction Learning sentence representations is central to many natural language modeling applications. The aim of a model for this task is to learn fixed- length feature vectors that encode the seman- tic and syntactic properties of sentences. Deep learning techniques have shown promising per- formance on sentence modeling, via feedfor- ward neural networks (Huang et al., 2013), re- current neural networks (RNNs) (Hochreiter and Schmidhuber, 1997), convolutional neural net- works (CNNs) (Kalchbrenner et al., 2014; Kim, 2014; Shen et al., 2014), and recursive neural net- works (Socher et al., 2013). Most of these models are task-dependent: they are trained specifically for a certain task. However, these methods may be- come inefficient when we need to repeatedly learn sentence representations for a large number of dif- ferent tasks, because they may require retraining a new model for each individual task. In this paper, in contrast, we are primarily interested in learning generic sentence representations that can be used across domains. Several approaches have been proposed for learn- ing generic sentence embeddings. The paragraph- vector model of Le and Mikolov (2014) incorpo- rates a global context vector into the log-linear neu- ral language model (Mikolov et al., 2013) to learn the sentence representation; however, at predic- tion time, one needs to perform gradient descent to compute a new vector. The sequence autoencoder of Dai and Le (2015) describes an encoder-decoder model to reconstruct the input sentence, while the skip-thought model of Kiros et al. (2015) extends the encoder-decoder model to reconstruct the sur- rounding sentences of an input sentence. Both the encoder and decoder of the methods above are mod- eled as RNNs. CNNs have recently achieved excellent results in various task-dependent natural language applica- tions as the sentence encoder (Kalchbrenner et al., 2014; Kim, 2014; Hu et al., 2014). This motivates us to propose a CNN encoder for learning generic …

Content truncated to 500 words