Parameter-free Sentence Embedding via Orthogonal Basis

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 638–648, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 638 Parameter-free Sentence Embedding via Orthogonal Basis Ziyi Yang1⇤, Chenguang Zhu2, and Weizhu Chen3 1Department of Mechanical Engineering, Stanford university 2Microsoft Speech and Dialogue Research Group 3Microsoft Dynamics 365 AI ziyi.yang@stanford.edu, {chezhu, wzchen}@microsoft.com Abstract We propose a simple and robust non- parameterized approach for building sentence representations. Inspired by the Gram- Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence. We model the semantic meaning of a word in a sentence based on two aspects. One is its relatedness to the word vector sub- space already spanned by its contextual words. The other is the word’s novel semantic mean- ing which shall be introduced as a new ba- sis vector perpendicular to this existing sub- space. Following this motivation, we develop an innovative method based on orthogonal ba- sis to combine pre-trained word embeddings into sentence representations. This approach requires zero parameters, along with efficient inference performance. We evaluate our ap- proach on 11 downstream NLP tasks. Our model shows superior performance compared with non-parameterized alternatives and it is competitive to other approaches relying on ei- ther large amounts of labelled data or pro- longed training time. 1 Introduction The concept of word embeddings has been preva- lent in NLP community in recent years, as they can characterize semantic similarity between any pair of words, achieving promising results in a large number of NLP tasks (Mikolov et al., 2013; Pen- nington et al., 2014; Salle et al., 2016). However, due to the hierarchical nature of human language, it is not sufficient to comprehend text solely based on isolated understanding of each word. This has prompted a recent rise in search for semantically robust embeddings for longer pieces of text, such as sentences and paragraphs. ⇤ Most of the work was done during internship at Mi- crosoft Based on learning paradigms, the existing ap- proaches to sentence embeddings can be catego- rized into two categories: i) parameterized meth- ods and ii) non-parameterized methods. Parameterized sentence embeddings. These models are parameterized and require training to optimize their parameters. SkipThought (Kiros et al., 2015) is an encoder-decoder model that pre- dicts adjacent sentences. Pagliardini et al. (2018) proposes an unsupervised model, Sent2Vec, to learn an n-gram feature in a sentence to predict the center word from the surrounding context. Quick thoughts (QT) (Logeswaran and Lee, 2018) re- places the encoder with a classifier to predict con- text sentences from candidate sequences. Khodak et al. (2018) proposes `a la carte to learn a linear mapping to reconstruct the center word from its context. Conneau et al. (2017) generates the sen- tence encoder InferSent using Natural Language Inference (NLI) dataset. Universal Sentence En- coder (Yang et al., 2018; Cer et al., 2018) uti- lizes the …

Content truncated to 500 words