A Connectionist Model of Anticipation in Visual Worlds

A Connectionist Model of Anticipation in Visual Worlds Marshall R. Mayberry, III, Matthew W. Crocker, and Pia Knoeferle Department of Computational Linguistics, Saarland University, Saarbr¨ucken, Germany {martym, crocker, knoferle}@coli.uni-sb.de Abstract. Recent “visual worlds” studies, wherein researchers study language in context by monitoring eye-movements in a visual scene during sentence process- ing, have revealed much about the interaction of diverse information sources and the time course of their influence on comprehension. In this study, five experi- ments that trade off scene context with a variety of linguistic factors are modelled with a Simple Recurrent Network modified to integrate a scene representation with the standard incremental input of a sentence. The results show that the model captures the qualitative behavior observed during the experiments, while retain- ing the ability to develop the correct interpretation in the absence of visual input. 1 Introduction There are two prevalent theories of language acquisition. One view emphasizes syntac- tic and semantic bootstrapping during language acquisition that enable children to learn abstract concepts from mappings between different kinds of information sources [1,2]. Another view emerges from connectionist literature and emphasizes the learning of lin- guistic structure from purely distributional properties of language usage [3,4]. While the perspectives are often taken to be diametrically opposed, both can be seen as crucially relying on correlations between words and their immediate context, be it the sentence as a whole or extra-linguistic input, such as a scene. We combine insights from both distributional and bootstrapping accounts in mod- elling the on-line comprehension of utterances in both the absence and presence of a vi- sual scene. This is an important achievement in at least two regards. First, it emphasizes the complementarity between distributional and bootstrapping approaches–discovering structure across linguistic and scene contexts [5]. Further, it is an important first step in linking situated models of on-line utterance comprehension more tightly to accounts of language acquisition, thus emphasizing the continuity of language processing. We present results from two simulations on a Simple Recurrent Network (SRN; [3]). Modification of the network to integrate input from a scene together with the charac- teristic incremental processing of such networks allowed us to model people’s ability to adaptively use the contextual information in order to more rapidly interpret and dis- ambiguate a sentence. The model draws on recent studies that appeal to theories of lan- guage acquisition to account for the comprehension of scene-related utterances [6,7]. Recent research within the visual worlds paradigm, wherein participants’ gazes to a scene while listening to an utterance are monitored, provides support for this view. Find- ings from this paradigm support an account of scene-related utterance comprehension in R. Dale et al. (Eds.): IJCNLP 2005, LNAI 3651, pp. 849–861, 2005. c⃝ Springer-Verlag Berlin Heidelberg 2005 850 M.R. Mayberry, III, M.W. Crocker, and P. Knoeferle which the rapid coordinated interaction of information from the immediate scene, and linguistic knowledge plays a major role in incremental and anticipatory comprehension. 2 Simulation 1 In Simulation 1, we simultaneously model four experiments that show the …

Content truncated to 500 words