Learning distributed sentence vectors with bi-directional 3D convolutions

Proceedings of the 28th International Conference on Computational Linguistics, pages 6820–6830 Barcelona, Spain (Online), December 8-13, 2020 6820 Learning distributed sentence vectors with bi-directional 3D convolutions Bin Liu, Liang Wang Center of Statistical Research, School of Statistics, Southwestern University of Finance & Economics Chengdu, China {liubin@,wangliang@smail}swufe.edu.cn Guosheng Yin Department of Statistics and Actuarial Science, The University of Hong Kong Hong Kong, China gyin@hku.hk Abstract We propose to learn distributed sentence representation using the text’s visual features as in- put. Different from the existing methods that render the words (or characters) of a sentence into images separately, we fold these images into a 3-dimensional sentence tensor. Then, multiple 3- dimensional convolutions with different lengths (the third dimension) are applied to the sentence tensor, which would act as bi-gram, tri-gram, quad-gram, and even five-gram detectors jointly. Similar to the Bi-LSTMs, these n-gram detectors learn both forward and backward distributional semantic knowledge from the sentence tensor. The proposed model uses bi-directional convolu- tions to learn text embedding according to the semantic order of words. The feature maps from the two directions are concatenated for final sentence embedding learning. Our model involves only a single layer of convolution which makes it easy and fast to train. We evaluate the sentence embeddings on several downstream natural language processing (NLP) tasks, which demonstrate surprisingly excellent performance of the proposed model. 1 Introduction Mapping documents or sentences to vectors (Le and Mikolov, 2014; Pagliardini et al., 2018) is the foundation of various natural language processing (NLP) tasks, such as text classification (Kim, 2014), paraphrase detection (Socher et al., 2011), natural language inference (Bowman et al., 2015), question answering (Zhou et al., 2015), etc. The most straightforward approach to sentence representation uses the bag-of-words model that represents a sentence as a bag of its constituent words, disregarding grammar and even the word order but keeping multiplicity. Another similar approach is called Glove (Pennington et al., 2014), which takes the average of word vectors of the constituent words of a sentence. These approaches are typically efficient to train, while they ignore the sequential characteristic of the text. To account for the word order, Skip-Thought (Kiros et al., 2015) learns sentence representation in an unsupervised way inspired by the skip-gram. It aims to predict the neighboring sentences or phrases for a given sentence. However, the training process of the Skip-Thought is very slow, which motivates the FastSent (Kenter et al., 2016) to speed up the training by representing a sentence as a sum of its constituent word vectors. Although FastSent is faster than Skip-Thought in training, it sacrifices the order of words in a sentence, which is important in language models, such as the n-gram feature. For example, Gupta et al. (2019) utilize the bi-gram and even tri-gram to train their embedding model. As discussed above, most existing works of sentence representation require pre-trained word vectors as the input or initialization. Sentence representation is taken as a downstream task of word representation. However, when human beings read a sentence or …

Content truncated to 500 words