Convolutional Neural Network for Universal Sentence Embeddings

Proceedings of the 27th International Conference on Computational Linguistics, pages 2470–2481 Santa Fe, New Mexico, USA, August 20-26, 2018. 2470 Convolutional Neural Network for Universal Sentence Embeddings Xiaoqi Jiao1 Fang Wang1,2 Dan Feng1 1Wuhan National Laboratory for Optoelectronics, Key Laboratory of Information Storage System, (School of Computer Science and Technology, Huazhong University of Science and Technology), Ministry of Education of China 2Shenzhen Huazhong University of Science and Technology Research Institute Corresponding author: wangfang@hust.edu.cn Abstract This paper proposes a simple CNN model for creating general-purpose sentence embeddings that can transfer easily across domains and can also act as effective initialization for downstream tasks. Recently, averaging the embeddings of words in a sentence has proven to be a surprisingly successful and efficient way of obtaining sentence embeddings. However, these models represent a sentence, only in terms of features of words or uni-grams in it. In contrast, our model (CSE) uti- lizes both features of words and n-grams to encode sentences, which is actually a generalization of these bag-of-words models. The extensive experiments demonstrate that CSE performs better than average models in transfer learning setting and exceeds the state of the art in supervised learning setting by initializing the parameters with the pre-trained sentence embeddings. 1 Introduction Representing word sequences such as phrases and sentences plays an important role in natural language understanding systems. In recent years, many composition functions have been applied to word em- beddings to obtain vectors for longer phrases or sentences, ranging from simple operations like addition (Mitchell and Lapata, 2008) to richly-structured functions like recursive neural networks (Socher et al., 2011), convolutional neural networks (Kalchbrenner et al., 2014), and recurrent neural networks (Tai et al., 2015). In this paper, based on convolutional neural networks (CNN), we introduce an architecture that can use both features of words and features of n-grams to encode sentences into real-valued vectors with the property that sentences with similar meaning have high cosine similarity in the embedding space. Our goal is to learn general-purpose sentence representations that can be transferred for measuring semantic textual similarity (STS) (Agirre et al., 2012) and can also act as effective initialization for downstream tasks. Originally, the CNN model is invented to extract local features in computer vision (Lecun et al., 1998). But it has been shown to be effective for NLP and has achieved excellent results in sentence modeling (Kalchbrenner et al., 2014), and other traditional NLP tasks (Collobert et al., 2011). To encode the meaning of a sentence of varying length into a fix-length vector, our simple CNN model first utilizes multiple filters with different size to extract possible features of n-grams in it. Then we obtain one feature corresponding to one filter through pooling operation. The intuition is to capture one aspect of semantic features of n-grams for each filter. Inspired by the strong performance of simply averaging word embeddings (Wieting et al., 2016), we also set filter size to 1 to extract word semantic information. To evaluate our model, we consider two types of learning …

Content truncated to 500 words