Putting Words in Context: LSTM Language Models and Lexical Ambiguity

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3342–3348 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3342 Putting words in context: LSTM language models and lexical ambiguity Laura Aina Kristina Gulordava Universitat Pompeu FabraGemma Boleda Barcelona, Spain {firstname.lastname}@upf.edu Abstract In neural network models of language, words are commonly represented using context- invariant representations (word embeddings) which are then put in context in the hidden lay- ers. Since words are often ambiguous, repre- senting the contextually relevant information is not trivial. We investigate how an LSTM language model deals with lexical ambiguity in English, designing a method to probe its hidden representations for lexical and contex- tual information about words. We find that both types of information are represented to a large extent, but also that there is room for improvement for contextual information. 1 Introduction In language, a word can contribute a very differ- ent meaning to an utterance depending on the con- text, a phenomenon known as lexical ambiguity (Cruse, 1986; Small et al., 2013). This variation is pervasive and involves both morphosyntactic and semantic aspects. For instance, in the examples in Table 1, show is used as a verb in Ex. (1), and as a noun in Ex. (2-3), in a paradigmatic case of mor- phosyntactic ambiguity in English. Instead, the difference between Ex. (2) and (3) is semantic in nature, with show denoting a TV program and an exhibition, respectively. Semantic ambiguity cov- ers a broad spectrum of phenomena, ranging from quite distinct word senses (e.g. mouse as animal or computer device) to more subtle lexical modu- lation (e.g. visit a city / an aunt / a doctor; Cruse, 1986). This paper investigates how deep learning models of language, and in particular Long Short- Term Memory Networks (LSTMs) trained on Lan- guage Modeling, deal with lexical ambiguity.1 In neural network models of language, words in a sentence are commonly represented through 1Code at: https://github.com/amore-upf/ LSTM_ambiguity word-level representations that do not change across contexts, that is, “static” word embeddings. These are then passed to further processing lay- ers, such as the hidden layers in a recurrent neural network (RNN). Akin to classic distributional se- mantics (Erk, 2012), word embeddings are formed as an abstraction over the various uses of words in the training data. For this reason, they are apt to represent context-invariant information about a word —its lexical information— but not the con- tribution of a word in a particular context —its contextual information (Erk, 2010). Indeed, word embeddings subsume information relative to var- ious senses of a word (e.g., mouse is close to words from both the animal and computer do- main; Camacho-Collados and Pilehvar, 2018). Classic distributional semantics attempted to do composition to account for contextual effects, but it was in general unable to go beyond short phrases (Baroni, 2013); newer-generation neural network models have supposed a big step forward, as they can natively do composition (Westera and Boleda, 2019). In particular, the hidden …

Content truncated to 500 words