Deixis and Conjunction in Multimodal Systems

Deixis and Conjunction in Multimodal Systems Michael Johnston AT&T Labs - Research Shannon Laboratory, 180 Park Ave Florham Park, NJ 07932, USA johnston@research.att.com Abstract In order to realize their full potential, multimodal interfaces need to support not just input from multiple modes, but single commands optimally distributed across the available input modes. A multimodal language processing architecture is needed to integrate semantic content from the different modes. Johnston 1998a proposes a modular approach to multimodal language processing in which spoken language parsing is completed before multimodal parsing. In this paper, I will demonstrate the difficulties this approach faces as the spoken language parsing component is expanded to provide a compositional analysis of deictic expressions. I propose an alternative architecture in which spoken and multimodal parsing are tightly interleaved. This architecture greatly simplifies the spoken language parsing grammar and enables predictive information from spoken language parsing to drive the application of multimodal parsing and gesture combination rules. I also propose a treatment of deictic numeral expressions that supports the broad range of pen gesture combinations that can be used to refer to collections of objects in the interface. Introduction Multimodal interfaces allow content to be conveyed between humans and machines over multiple different channels such speech, graphics, pen, and hand gesture. This enables more natural and efficient interaction since different kinds of content are best suited to particular modes. For example, spatial information is effectively conveyed using gesture for input (Oviatt 1997) and 2d or 3d graphics for output (Towns et al 1998). Multimodal interfaces also stand to play a critical role in the ongoing migration of interaction onto wireless portable computing devices, such as PDAs and next generation phones, which have limited screen real estate and no keyboard. For such devices, complex graphical user interfaces are not feasible and speech and pen will be the primary input modes. I focus here on multimodal interfaces which support speech and pen input. Pen input consists of gestures and drawings which are made in electronic ink on the computer display and processed by a gesture recognizer. Speech input is transcribed using a speech recognizer. This paper is concerned with the relationship between spoken language parsing and multimodal parsing, specifically whether they should be separate modular components, and the related issue of determining the appropriate level of constituent structure at which multimodal integration should apply. Johnston 1998a proposes a modular approach in which the individual modes are parsed and assigned typed feature structures representing their combinatory properties and semantic content. A multidimensional chart parser then combines these structures in accordance with a unification-based multimodal grammar. This approach is outlined in Section 1. Section 2 addresses the compositional analysis of deictic expressions and their interaction with conjunction and other aspects of the grammar. In Section 3, a new architecture is presented in which spoken and multimodal parsing are interleaved. Section 4 presents an analysis of deictic numeral expressions, and Section 5 discusses certain constructions in which multimodal integration applies at higher levels of constituent structure …

Content truncated to 500 words