EigenSent: Spectral sentence embeddings using higher-order Dynamic Mode Decomposition

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4536–4546 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4536 EigenSent: Spectral sentence embeddings using higher-order Dynamic Mode Decomposition Subhradeep Kayal George Tsatsaronis Elsevier B.V. Amsterdam, NL. subhradeep.kayal@gmail.com Abstract Distributed representation of words, or word embeddings, have motivated methods for cal- culating semantic representations of word se- quences such as phrases, sentences and para- graphs. Most of the existing methods to do so either use algorithms to learn such repre- sentations, or improve on calculating weighted averages of the word vectors. In this work, we experiment with spectral methods of signal representation and summarization as mech- anisms for constructing such word-sequence embeddings in an unsupervised fashion. In particular, we explore an algorithm rooted in fluid-dynamics, known as higher-order Dy- namic Mode Decomposition, which is de- signed to capture the eigenfrequencies, and hence the fundamental transition dynamics, of periodic and quasi-periodic systems. It is em- pirically observed that this approach, which we call EigenSent, can summarize transitions in a sequence of words and generate an em- bedding that can represent well the sequence itself. To the best of the authors’ knowledge, this is the first application of a spectral decom- position and signal summarization technique on text, to create sentence embeddings. We test the efficacy of this algorithm in creating sentence embeddings on three public datasets, where it performs appreciably well. More- over it is also shown that, due to the positive combination of their complementary proper- ties, concatenating the embeddings generated by EigenSent with simple word vector averag- ing achieves state-of-the-art results. 1 Introduction 1.1 Relevant concepts Word embeddings are dense vectors that capture the semantic and contextual information of a word, and are ubiquitous in natural language processing tasks across many domains (Camacho-Collados and Pilehvar, 2018). Several different algorithms and models for constructing these embeddings have been proposed and evaluated in literature (Perone et al., 2018). A natural next step is to extend the notion of word embeddings to the level of a sentence (or paragraph, or document). Such representations are known as sentence embeddings, often inter- changeably used with the terms paragraph embed- dings or document embeddings, and should, ide- ally, capture the meaning of a sentence (Le and Mikolov, 2014). More recently, the concept of universal sen- tence embeddings has gained traction, as they leverage models trained on large text corpuses in a way which is task-agnostic. These pre-trained models can then be used in a wide array of down- stream tasks, often performing better in those tasks when little training data is available (Subramanian et al., 2018). 1.2 A brief review of literature Word embedding methods vary from complex neural language models (Bengio et al., 2003) and semi-supervised approaches (Turian et al., 2010), to simpler and faster methods such as Word2Vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), ELMo (Peters et al., 2018) and BERT (De- vlin et al., 2019), that can be trained on much …

Content truncated to 500 words