Retrofitting Contextualized Word Embeddings with Paraphrases

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1198–1203, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1198 Retrofitting Contextualized Word Embeddings with Paraphrases Weijia Shi1∗, Muhao Chen1,2∗, Pei Zhou1,3 and Kai-Wei Chang1 1Department of Computer Science, University of California Los Angeles 2Department of Computer and Information Science, University of Pennsylvania 3Department of Computer Science, University of Southern California {swj0419, kwchang}@cs.ucla.edu; muhao@seas.upenn.edu; peiz@usc.edu Abstract Contextualized word embedding models, such as ELMo, generate meaningful representations of words and their context. These models have been shown to have a great impact on downstream applications. However, in many cases, the contextualized embedding of a word changes drastically when the context is para- phrased. As a result, the downstream model is not robust to paraphrasing and other linguis- tic variations. To enhance the stability of con- textualized word embedding models, we pro- pose an approach to retrofitting contextualized embedding models with paraphrase contexts. Our method learns an orthogonal transforma- tion on the input space, which seeks to mini- mize the variance of word representations on paraphrased contexts. Experiments show that the retrofitted model significantly outperforms the original ELMo on various sentence classi- fication and language inference tasks. 1 Introduction Contextualized word embeddings have shown to be useful for a variety of downstream tasks (Pe- ters et al., 2018, 2017; McCann et al., 2017). Un- like traditional word embeddings that represent words with fixed vectors, these embedding mod- els encode both words and their contexts and gen- erate context-specific representations. While con- textualized embeddings are useful, we observe that a language model-based embedding model, ELMo (Peters et al., 2018), cannot accurately cap- ture the semantic equivalence of contexts. Specif- ically, in cases where the contexts of a word have equivalent or similar meanings but are changed in sentence formation or word order, ELMo may assign very different representations to the word. Table 1 shows two examples, where ELMo gen- erates very different representations for the bold- faced words under semantic equivalent contexts. ∗ Both authors contributed equally to this work. Paraphrased contexts L2 Cosine How can I make bigger my arms? How do I make my arms bigger? 6.42 0.27 Some people believe earth is flat. Why? Why do people still believe in flat earth? 7.59 0.46 It is a very small window. I have a large suitcase. 5.44 0.26 Table 1: L2 and Cosine distances between embeddings of boldfaced words. The distance between the shared word in the paraphrases is even greater than the dis- tance between large and small in random contexts. . Quantitatively, 28.3% of the shared words in the paraphrase sentence pairs on the MRPC corpus (Dolan et al., 2004) is larger than the average dis- tance between good and bad in random contexts, and 41.5% of those exceeds the distance between large and small. As a result, the downstream model is not robust to paraphrasing and the per- formance is hindered. Infusing …

Content truncated to 500 words