Like a Baby: Visually Situated Neural Language Acquisition

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5127–5136 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5127 Like a Baby: Visually Situated Neural Language Acquisition Alexander G. Ororbia*1,2, Ankur Mali *1,2, Matthew A. Kelly1, and David Reitter1,3 (1) The Pennsylvania State University, University Park, PA, USA (2) Rochester Institute of Technology, Rochester, NY, USA (3) Google Research, New York City, NY, USA ago@cs.rit.edu, aam35@psu.edu, matthew.kelly@psu.edu, reitter@google.com Abstract We examine the benefits of visual context in training neural language models to perform next-word prediction. A multi-modal neural architecture is introduced that outperform its equivalent trained on language alone with a 2% decrease in perplexity, even when no vi- sual context is available at test. Fine-tuning the embeddings of a pre-trained state-of-the- art bidirectional language model (BERT) in the language modeling framework yields a 3.5% improvement. The advantage for train- ing with visual context when testing without is robust across different languages (English, German and Spanish) and different models (GRU, LSTM, ∆-RNN, as well as those that use BERT embeddings). Thus, language mod- els perform better when they learn like a baby, i.e, in a multi-modal environment. This find- ing is compatible with the theory of situated cognition: language is inseparable from its physical context. 1 Introduction The theory of situated cognition postulates that a person’s knowledge is inseparable from the physi- cal or social context in which it is learned and used (Greeno and Moore, 1993). Similarly, Perceptual Symbol Systems theory holds that all of cogni- tion, thought, language, reasoning, and memory, is grounded in perceptual features (Barsalou, 1999). Knowledge of language cannot be separated from its physical context, which allows words and sen- tences to be learned by grounding them in refer- ence to objects or natural concepts on hand (see Roy and Reiter, 2005, for a review). Nor can knowledge of language be separated from its so- cial context, where language is learned interac- tively through communicating with others to facil- itate problem-solving. Simply put, language does not occur in a vacuum. Yet, statistical language models, typically con- nectionist systems, are often trained in such a vac- uum. Sequences of symbols, such as sentences or phrases composed of words in any language, such as English or German, are often fed into the model independently of any real-world con- text they might describe. In the classical language modeling framework, a model learns to predict a word based on a history of words it has seen so far. While these models learn a great deal of linguis- tic structure from these symbol sequences alone, acquiring the essence of basic syntax, it is highly unlikely that this approach can create models that acquire much in terms of semantics or pragmat- ics, which are integral to the human experience of language. How might one build neural lan- guage models that “understand” the semantic con- tent held within the symbol sequences, of any lan- guage, presented to it? In this paper, we take …

Content truncated to 500 words