A Model for the Cross-Modal Influence of Visual Context upon Language Procesing

International Conference RANLP 2009 - Borovets, Bulgaria, pages 230–235 A Model for the Cross-Modal Influence of Visual Context upon Language Processing Patrick McCrae CINACS Graduate Research Group Department of Informatics Hamburg University Vogt-K¨olln-Straße 30, 22527 Hamburg, Germany patrick.mccrae@informatik.uni-hamburg.de Abstract In this paper, we present a novel, cognitively mo- tivated framework for modelling the cross-modal influence of visual scene context upon language processing. We illustrate how semantic relations in a knowledge representation of visual scene context can effect syntactic attachment modulations in a weighted- constraint dependency parser. In line with a central tenet of conceptual semantics, visual scene context and linguistic processing are hypothesised to interact via an intermediate, cross-modally integrated level of semantic representation. Cross-modal interaction in our model is restricted by conceptual compatibility between the concepts activated linguistically and contextually. We apply our framework to syntactically ambiguous sentences of German and parse them in the presence of biasing visual scene contexts. The observed modulations in syntactic attachment support our two modelling hypotheses: 1) The influence of visual context upon syntactic processing is mediated by semantics. 2) The compatibility of concepts from different modalities is a suitable criterion to restrict the scope of cross-modal interaction. Keywords Cross-Modal Interaction, Parsing, Syntax-Semantics Interface, Context Integration, Information Fusion. 1 Introduction The vision-language interface has become considerably more accessible to scientific enquiry with the advent of eye tracking technology. [3] showed a semantic interaction between vision and language for single-word processing with co-present visual stimuli as early as 1974. About two decades later, [13] investigated subjects’ eye movements for syntactically ambiguous sentences with a particular fo- cus on the aspect of incrementality in linguistic processing. Yet another decade later, [1] investigated whether the eye movement patterns observed as a result of the interaction between vision and language were contingent upon the vi- sual stimulus being co-present with the linguistic stimulus. In this paper we present a successful implementation of a framework for the integration of visual context infor- mation into the process of syntactic parsing. Starting from the review of central empirical investigations of the vision-language interface, we begin with the identifica- tion of elementary requirements for the design of a cog- nitively motivated framework for the cross-modal integra- tion between vision and language. Adopting a weighted- constraint model of language processing, we outline how in our framework the interpretation of visual context con- stitutes an additional constraint on the cross-modally inte- grated semantic representation which is built up based on linguistic and contextual input. In the following section, we provide a brief overview over selected key findings from milestone experiments at the vision-language interface to motivate the requirements for our framework. In Section 3, we outline how the compo- nents of our framework interact with each other and which procedures we employ to achieve cross-modal interaction. In Section 4, we provide experimental results from the in- tegration of visual context into parsing for a particular class of syntactically ambiguous sentences. In Section 5, we summarise our central points and draw conclusions. 2 Milestone …

Content truncated to 500 words