Specializing Word Embeddings (for Parsing) by Information Bottleneck

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2744–2754, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2744 Specializing Word Embeddings (for Parsing) by Information Bottleneck Xiang Lisa Li Department of Computer Science Johns Hopkins University xli150@jhu.edu Jason Eisner Department of Computer Science Johns Hopkins University jason@cs.jhu.edu Abstract Pre-trained word embeddings like ELMo and BERT contain rich syntactic and semantic in- formation, resulting in state-of-the-art perfor- mance on various tasks. We propose a very fast variational information bottleneck (VIB) method to nonlinearly compress these embed- dings, keeping only the information that helps a discriminative parser. We compress each word embedding to either a discrete tag or a continuous vector. In the discrete version, our automatically compressed tags form an alter- native tag set: we show experimentally that our tags capture most of the information in tra- ditional POS tag annotations, but our tag se- quences can be parsed more accurately at the same level of tag granularity. In the continu- ous version, we show experimentally that mod- erately compressing the word embeddings by our method yields a more accurate parser in 8 of 9 languages, unlike simple dimensionality reduction. 1 Introduction Word embedding systems like BERT and ELMo use spelling and context to obtain contextual em- beddings of word tokens. These systems are trained on large corpora in a task-independent way. The resulting embeddings have proved to then be useful for both syntactic and semantic tasks, with different layers of ELMo or BERT being somewhat special- ized to different kinds of tasks (Peters et al., 2018b; Goldberg, 2019). State-of-the-art performance on many NLP tasks can be obtained by fine-tuning, i.e., back-propagating task loss all the way back into the embedding function (Peters et al., 2018a; Devlin et al., 2018). In this paper, we explore what task-specific in- formation appears in the embeddings before fine- tuning takes place. We focus on the task of de- pendency parsing, but our method can be easily sentence w embeddings x taggings t tree y decoder encoder ELMo Ollivander sold the wand . dobj subj det punct Parser Figure 1: Our instantiation of the information bottle- neck, with bottleneck variable T. A jagged arrow indi- cates a stochastic mapping, i.e. the jagged arrow points from the parameters of a distribution to a sample drawn from that distribution. extended to other syntactic or semantic tasks. Our method compresses the embeddings by extracting just their syntactic properties—specifically, the in- formation needed to reconstruct parse trees (be- cause that is our task). Our nonlinear, stochas- tic compression function is explicitly trained by variational information bottleneck (VIB) to forget task-irrelevant information. This is reminiscent of canonical correspondence analysis (Anderson, 2003), a method for reducing the dimensionality of an input vector so that it remains predictive of an output vector, although we are predicting an out- put tree instead. However, VIB goes beyond mere dimensionality reduction to a fixed lower dimen- sionality, …

Content truncated to 500 words