Scaffolded input promotes atomic organization in the recurrent neural network language model

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 408–422 November 10–11, 2021. ©2021 Association for Computational Linguistics 408 Scaffolded input promotes atomic organization in the recurrent neural network language model Philip A. Huebner and Jon A. Willits Department of Psychology University of Illinois at Urbana-Champaign {huebner3, jwillits}@illinois.edu Abstract The recurrent neural network (RNN) language model is a powerful tool for learning arbitrary sequential dependencies in language data. De- spite its enormous success in representing lexi- cal sequences, little is known about the qual- ity of the lexical representations that it ac- quires. In this work, we conjecture that it is straightforward to extract lexical represen- tations (i.e. static word embeddings) from an RNN, but that the amount of semantic infor- mation that is encoded is limited when lexi- cal items in the training data provide redun- dant semantic information. We conceptualize this limitation of the RNN as a failure to learn atomic internal states - states which capture in- formation relevant to single word types with- out being influenced by redundant information provided by words with which they co-occur. Using a corpus of artificial language, we ver- ify that redundancy in the training data yields non-atomic internal states, and propose a novel method for inducing atomic internal states. We show that 1) our method successfully induces atomic internal organization in controlled ex- periments, and 2) under more realistic condi- tions in which the training consists of child- directed language, application of our method improves the performance of lexical represen- tations on a downstream semantic categoriza- tion task. 1 Introduction In order to reproduce the full range of human lan- guage behavior, a language system needs to be able to represent sequences of words, while simul- taneously being capable of forming semantically organized representations of individual words. To date, researchers have had difficulty specifying a model or system that is good at both tasks. On the one hand, static word embedding models are better at representing individual word meanings (Mikolov et al., 2013), but do not naturally lend themselves to multi-word sequences. On the other hand, language models based on recurrent neural networks (Elman, 1990) or Transformers (Vaswani et al., 2017) are well-suited for dealing with word sequences, but usually perform worse than static word embedding models on word-level tasks (Huebner and Willits, 2018) and are challenging to extract word embed- dings from (Vuli´c et al., 2020). In this paper, we diagnose why recurrent neu- ral networks (RNNs) learn sub-optimal represen- tations of individual words. In short, we argue that the language modeling objective encodes in- formation relevant to individual words by spread- ing it across representations of neighboring words when they provide partially redundant information, a phenomenon related to entanglement (Bengio et al., 2013). Although this supports prediction of words in context, this impedes the formation of stand-alone lexical representations that are useful for tasks requiring de-contextualized information about individual words. In this work, we propose a simple strategy for protecting against this kind of context-sensitivity, …

Content truncated to 500 words