Deep Contextualized Word Representations

Proceedings of NAACL-HLT 2018, pages 2227–2237 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Deep contextualized word representations Matthew E. Peters†, Mark Neumann†, Mohit Iyyer†, Matt Gardner†, {matthewp,markn,mohiti,mattg}@allenai.org Christopher Clark⇤, Kenton Lee⇤, Luke Zettlemoyer†⇤ {csquared,kentonl,lsz}@cs.washington.edu †Allen Institute for Artificial Intelligence ⇤Paul G. Allen School of Computer Science & Engineering, University of Washington Abstract We introduce a new type of deep contextual- ized word representation that models both (1) complex characteristics of word use (e.g., syn- tax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned func- tions of the internal states of a deep bidirec- tional language model (biLM), which is pre- trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, tex- tual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals. 1 Introduction Pre-trained word representations (Mikolov et al., 2013; Pennington et al., 2014) are a key compo- nent in many neural language understanding mod- els. However, learning high quality representa- tions can be challenging. They should ideally model both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). In this paper, we introduce a new type of deep contextualized word representation that directly addresses both challenges, can be easily integrated into existing models, and significantly improves the state of the art in every considered case across a range of challenging language un- derstanding problems. Our representations differ from traditional word type embeddings in that each token is assigned a representation that is a function of the entire input sentence. We use vectors derived from a bidirec- tional LSTM that is trained with a coupled lan- guage model (LM) objective on a large text cor- pus. For this reason, we call them ELMo (Em- beddings from Language Models) representations. Unlike previous approaches for learning contextu- alized word vectors (Peters et al., 2017; McCann et al., 2017), ELMo representations are deep, in the sense that they are a function of all of the in- ternal layers of the biLM. More specifically, we learn a linear combination of the vectors stacked above each input word for each end task, which markedly improves performance over just using the top LSTM layer. Combining the internal states in this manner al- lows for very rich word representations. Using in- trinsic evaluations, we show that the higher-level LSTM states capture context-dependent aspects of word meaning (e.g., they can be used with- out modification to perform well on supervised word sense disambiguation tasks) while lower- level states model aspects of syntax (e.g., they can be used to do part-of-speech tagging). Simultane- ously exposing all of these signals is highly …

Content truncated to 500 words