Using Eye Movements to Study Spoken Language Comprehension: Evidence for Incremental Interpretation (Invited Talk)

INVITED TALK Eye Movements and Spoken Language Comprehension Michael K. Tanenhaus* Department of Brain and Cognitive Sciences University of Rochester Rochester, NY 14627 mtan@bcs, rochester, edu Michael J. Spivey-Knowlton Department of Psychology Cornell University Ithaca, NY 14583 mj sk@cornel i. edu Kathleen M. Eberhard Department of Psychology University of Notre Dame Notre Dame, IN 46556 kathleen .m. eberhard, l@nd. edu Julie C. Sedivy Department of Linguistics University of Rochester Rochester, NY 14627 sedivy@bcs, rochester, edu Paul D. Allopenna Department of Brain and Cognitive Sciences University of Rochester Rochester, NY 14627 al lopenObcs, rochester, edu James S. Magnuson Department of Brain and Cognitive Sciences University of Rochester Rochester, NY 14627 magnuson@bcs, rochester, edu Abstract We present an overview of recent work in which eye movements are monitored as people follow spoken instructions to move objects or pictures in a visual workspace. Subjects naturally make saccadic eye-movements to objects that are closely time-locked to relevant information in the instruction. Thus the eye-movements provide a window into the rapid mental processes that underlie spoken language comprehension. We review studies of reference resolution, word recognition, and pragmatic effects on syntactic ambiguity resolution. Our studies show that people seek to establish reference with respect to their behavioral goals during the earliest moments of linguistic processing. Moreover, referentially relevant non-linguistic information immediately affects how the linguistic input is initially structured. Introduction Many important questions about language comprehension can only be answered by examining processes that are closely time-locked to the linguistic input. These processes take place quite rapidly and they are largely opaque to introspection. As a consequence, psycholinguists have increasingly turned to experimental methods designed to tap real-time language processing. These include a variety of reading time measures as well as paradigms in which subjects monitor the incoming speech for targets or respond to visually presented probes. The hope is that these "on- line" measures can provide information that can be used to inform and evaluate explicit computational models of language processing. Although on-line measures have provided increasingly fine-grained information about the time-course of language processing, they ,are also limited in some important respects. Perhaps the most serious limitation is that they cannot be used to study language in natural tasks with real-world referents. This makes it difficult to study how interpretation develops. Moreover, the emphasis on processing "decontextualized" language may be underestimating the importance of interpretive processes in immediate language processing. Recently, we have been exploring a new paradigm for studying spoken language comprehension. Participants in our experiments follow spoken instructions to touch or manipulate objects in a visual workspace while we monitor their eye-movements using a lightweight camera mounted on a headband. The camera, manufactured by Applied Scientific Laboratories, provides an infrared image of the eye at 60Hz. The center of the pupil and the corneal reflection are tracked to determine the orbit of the eye relative to file head. Accuracy is better than one degree of arc, with virtually unrestricted head and body movements [Ballard, Hayhoe, and Pelz, 1995]. Instructions are spoken …

Content truncated to 500 words