Efficient Sentence Embedding using Discrete Cosine Transform

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3672–3678, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3672 Efficient Sentence Embedding using Discrete Cosine Transform Nada Almarwani,∗1,2 Hanan Aldarmaki,∗3 Mona Diab 1,4 1 Dept. of Computer Science, The George Washington University 2 Dept. of Computer Science, Taibah University 3 Computer Science & Software Engineering Dept., UAEU 4 AWS, Amazon AI nadaoh@gwu.edu, h-aldarmaki@uaeu.ac.ae, diabmona@amazon.com Abstract Vector averaging remains one of the most pop- ular sentence embedding methods in spite of its obvious disregard for syntactic structure. While more complex sequential or convolu- tional networks potentially yield superior clas- sification performance, the improvements in classification accuracy are typically mediocre compared to the simple vector averaging. As an efficient alternative, we propose the use of discrete cosine transform (DCT) to compress word sequences in an order-preserving man- ner. The lower order DCT coefficients repre- sent the overall feature patterns in sentences, which results in suitable embeddings for tasks that could benefit from syntactic features. Our results in semantic probing tasks demonstrate that DCT embeddings indeed preserve more syntactic information compared with vector averaging. With practically equivalent com- plexity, the model yields better overall perfor- mance in downstream classification tasks that correlate with syntactic features, which illus- trates the capacity of DCT to preserve word order information. 1 Introduction Modern NLP systems rely on word embeddings as input units to encode the statistical semantic and syntactic properties of words, ranging from standard context-independent embeddings such as word2vec (Mikolov et al., 2013) and Glove (Pen- nington et al., 2014) to contextualized embeddings such as ELMo (Peters et al., 2018) and BERT (De- vlin et al., 2018). However, most applications op- erate at the phrase or sentence level. Hence, the word embeddings are averaged to yield sentence embeddings. Averaging is an efficient composi- tional operation that leads to good performance. In fact, averaging is difficult to beat by more complex ∗Both authors contributed equally. compositional models as illustrated across several classification tasks: topic categorization, semantic textual similarity, and sentiment classification (Al- darmaki and Diab, 2018). Encoding sentences into fixed-length vectors that capture various full sen- tence linguistic properties leading to performance gains across different classification tasks remains a challenge. Given the complexity of most mod- els that attempt to encode sentence structure, such as convolutional, recursive, or recurrent networks, the trade-off between efficiency and performance tips the balance in favor of simpler models like vector averaging. Sequential neural sentence en- coders, like Skip-thought (Kiros et al., 2015) and InferSent (Conneau et al., 2017), can potentially encode rich semantic and syntactic features from sentence structures. However, for practical ap- plications, sequential models are rather cumber- some and inefficient, and the gains in performance are typically mediocre compared with vector av- eraging (Aldarmaki and Diab, 2018). In addition, the more complex models typically don’t gener- alize well to out-of-domain data (Wieting et al., 2015). FastSent (Hill et al., 2016) is …

Content truncated to 500 words