Learning Efficient Task-Specific Meta-Embeddings with Word Prisms

Proceedings of the 28th International Conference on Computational Linguistics, pages 1229–1241 Barcelona, Spain (Online), December 8-13, 2020 1229 Learning Efficient Task-Specific Meta-Embeddings with Word Prisms Jingyi He∗1, KC Tsiolis∗1, Kian Kenyon-Dean†∗2, Jackie C. K. Cheung1 Mila – Qu´ebec AI Institute / McGill University, Montr´eal, QC, Canada 1 BMO AI Capabilities Team – Bank of Montr´eal, Toronto, ON, Canada 2 {jingyi.he,kc.tsiolis}@mail.mcgill.ca, jcheung@cs.mcgill.ca kian.kenyon-dean@bmo.com Abstract Word embeddings are trained to predict word cooccurrence statistics, which leads them to pos- sess different lexical properties (syntactic, semantic, etc.) depending on the notion of context defined at training time. These properties manifest when querying the embedding space for the most similar vectors, and when used at the input layer of deep neural networks trained to solve downstream NLP problems. Meta-embeddings combine multiple sets of differently trained word embeddings, and have been shown to successfully improve intrinsic and extrinsic performance over equivalent models which use just one set of source embeddings. We introduce word prisms: a simple and efficient meta-embedding method that learns to combine source embeddings ac- cording to the task at hand. Word prisms learn orthogonal transformations to linearly combine the input source embeddings, which allows them to be very efficient at inference time. We eval- uate word prisms in comparison to other meta-embedding methods on six extrinsic evaluations and observe that word prisms offer improvements in performance on all tasks.1 1 Introduction A popular approach to representing word meaning in NLP is to characterize a word by “the company that it keeps” (Firth, 1957). This intuition is the basis of famous word embedding techniques such as Word2vec (Mikolov et al., 2013a) and Glove (Pennington et al., 2014). However, the question of what company a word keeps — i.e., what should define a word’s context — is open. A word’s context could be defined via a symmetric window of 1, 2, 5, 10, 20 words, the words that precede it, the words that follow it, the words with which it shares a dependency edge, etc. Determining the utility of such different notions of context for training word embeddings is a problem that has attracted considerable attention (Yatbaz et al., 2012; Levy and Goldberg, 2014a; Bansal et al., 2014; Lin et al., 2015; Melamud et al., 2016; Lison and Kutuzov, 2017) but there is no conclusive evidence that any single notion of context could be the best for solving NLP problems in general. Thus, many deep learning solutions for NLP have yet another hyperparameter to tune: what set of word embeddings should be selected for the input layer of the model. As NLP tasks become more and more complex, the practice of providing a deep model with only one notion of a word’s meaning becomes limiting. Word meta-embeddings address aspects of this problem by proposing techniques for combining multi- ple sets of word embeddings before providing them into the input layer of a downstream model. Yin and Sch¨utze (2016) motivated word meta-embeddings by arguing that they are advantageous for the follow- ing reasons: diversity — …

Content truncated to 500 words