Unification-based Multimodal Parsing

Unification-based Multimodal Parsing Michael Johnston Center for Human Computer Communication Department of Computer Science and Engineering Oregon Graduate Institute P.O. Box 91000, Portland, OR 97291-1000 johnston@cse.ogi.edu Abstract In order to realize their full potential, multimodal systems need to support not just input from multiple modes, but also synchronized integration of modes. Johnston et al (1997) model this integration using a unification opera- tion over typed feature structures. This is an effective so- lution for a broad class of systems, but limits multimodal utterances to combinations of a single spoken phrase with a single gesture. We show how the unification-based ap- proach can be scaled up to provide a full multimodal grammar formalism. In conjunction with a multidimen- sional chart parser, this approach supports integration of multiple elements distributed across the spatial, temporal, and acoustic dimensions of multimodal interaction. In- tegration strategies are stated in a high level unification- based rule formalism supporting rapid prototyping and it- erative development of multimodal systems. 1 Introduction Multimodal interfaces enable more natural and effi- cient interaction between humans and machines by providing multiple channels through which input or output may pass. Our concern here is with multi- modal input, such as interfaces which support simul- taneous input from speech and pen. Such interfaces have clear task performance and user preference ad- vantages over speech only interfaces, in particular for spatial tasks such as those involving maps (Ovi- att 1996). Our focus here is on the integration of in- put from multiple modes and the role this plays in the segmentation and parsing of natural human input. In the examples given here, the modes are speech and pen, but the architecture described is more general in that it can support more than two input modes and modes of other types such as 3D gestural input. Our multimodal interface technology is imple- mented in QuickSet (Cohen et al 1997), a work- ing system which supports dynamic interaction with maps and other complex visual displays. The initial applications of QuickSet are: setting up and inter- acting with distributed simulations (Courtemanche and Cercanowicz 1995), logistics planning, and nav- igation in virtual worlds. The system is distributed; consisting of a series of agents (Figure 1) which communicate through a shared blackboard (Cohen et al 1994). It runs on both desktop and handheld PCs, communicating over wired and wireless LANs. The user interacts with a map displayed on a wireless hand-held unit (Figure 2). Figure 1: Multimodal Architecture , U~kS.I t 21~ Beta 3 : S ¢*~a l[!~/uo,b ..,7.~ L22;.L . :'''.L':: ' ,~=~y,t 0 0A ~ d ~ 0oAl n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E Figure 2: User Interface They can draw directly on the map and simultane- ously issue spoken commands. Different kinds of entities, lines, and areas may be created by drawing the appropriate spatial features and speaking their type; for example, …

Content truncated to 500 words