Finite-state Multimodal Parsing and Understanding

Finite-state Multimodal Parsing and Understanding Michael Johnston AT&T Labs - Research Shannon Laboratory, 180 Park Ave Florham Park, NJ 07932, USA johnston@research.att.com Srinivas Bangalore AT&T Labs - Research Shannon Laboratory, 180 Park Ave Florham Park, NJ 07932, USA srini@research.att.com Abstract Multimodal interfaces require effective parsing and understanding of utterances whose content is dis- tributed across multiple input modes. Johnston 1998 presents an approach in which strategies for mul- timodal integration are stated declaratively using a unification-based grammar that is used by a multi- dimensional chart parser to compose inputs. This approach is highly expressive and supports a broad class of interfaces, but offers only limited potential for mutual compensation among the input modes, is subject to significant concerns in terms of computa- tional complexity, and complicates selection among alternative multimodal interpretations of the input. In this paper, we present an alternative approach in which multimodal parsing and understanding are achieved using a weighted finite-state device which takes speech and gesture streams as inputs and out- puts their joint interpretation. This approach is sig- nificantly more efficient, enables tight-coupling of multimodal understanding with speech recognition, and provides a general probabilistic framework for multimodal ambiguity resolution. 1 Introduction Multimodal interfaces are systems that allow input and/or output to be conveyed over multiple different channels such as speech, graphics, and gesture. They enable more natural and effective interaction since different kinds of content can be conveyed in the modes to which they are best suited (Oviatt, 1997). Our specific concern here is with multimodal inter- faces supporting input by speech, pen, and touch, but the approach we describe has far broader applicabil- ity. These interfaces stand to play a critical role in the ongoing migration of interaction from the desktop to wireless portable computing devices (PDAs, next- generation phones) that offer limited screen real es- tate, and other keyboard-less platforms such as pub- lic information kiosks. To realize their full potential, multimodal inter- faces need to support not just input from multiple modes, but synergistic multimodal utterances opti- mally distributed over the available modes (John- ston et al., 1997). In order to achieve this, an ef- fective method for integration of content from dif- ferent modes is needed. Johnston (1998b) shows how techniques from natural language processing (unification-based grammars and chart parsing) can be adapted to support parsing and interpretation of utterances distributed over multiple modes. In that approach, speech and gesture recognition produce n- best lists of recognition results which are assigned typed feature structure representations (Carpenter, 1992) and passed to a multidimensional chart parser that uses a multimodal unification-based grammar to combine the representations assigned to the input el- ements. Possible multimodal interpretations are then ranked and the optimal interpretation is passed on for execution. This approach overcomes many of the limitations of previous approaches to multimodal integration such as (Bolt, 1980; Neal and Shapiro, 1991) (See (Johnston et al., 1997)(p. 282)). It sup- ports speech with multiple gestures, visual parsing of unimodal gestures, and its declarative nature fa- cilitates rapid …

Content truncated to 500 words