Unification-based Multimodal Integration

Unification-based Multimodal Integration Michael Johnston, Philip R. Cohen, David McGee, Sharon L. Oviatt, James A. Pittman, Ira Smith Center for Human Computer Communication Department of Computer Science and Engineering Oregon Graduate Institute, PO BOX 91000, Portland, OR 97291, USA. {johnston, pcohen, dmcgee, oviatt, jay, ira}©cse, ogi. edu Abstract Recent empirical research has shown con- clusive advantages of multimodal interac- tion over speech-only interaction for map- based tasks. This paper describes a mul- timodal language processing architecture which supports interfaces allowing simulta- neous input from speech and gesture recog- nition. Integration of spoken and gestural input is driven by unification of typed fea- ture structures representing the semantic contributions of the different modes. This integration method allows the component modalities to mutually compensate for each others' errors. It is implemented in Quick- Set, a multimodal (pen/voice) system that enables users to set up and control dis- tributed interactive simulations. 1 Introduction By providing a number of channels through which information may pass between user and computer, multimodal interfaces promise to significantly in- crease the bandwidth and fluidity of the interface between humans and machines. In this work, we are concerned with the addition of multimodal input to the interface. In particular, we focus on interfaces which support simultaneous input from speech and pen, utilizing speech recognition and recognition of gestures and drawings made with a pen on a complex visual display, such as a map. Our focus on multimodal interfaces is motivated, in part, by the trend toward portable computing de- vices for which complex graphical user interfaces are infeasible. For such devices, speech and gesture will be the primary means of user input. Recent em- pirical results (Oviatt 1996) demonstrate clear task performance and user preference advantages for mul- timodal interfaces over speech only interfaces, in par- ticular for spatial tasks such as those involving maps. Specifically, in a within-subject experiment during which the same users performed the same tasks in various conditions using only speech, only pen, or both speech and pen-based input, users' multimodal input to maps resulted in 10% faster task comple- tion time, 23% fewer words, 35% fewer spoken dis- fluencies, and 36% fewer task errors compared to unimodal spoken input. Of the user errors, 48% in- volved location errors on the map--errors that were nearly eliminated by the simple ability to use pen- based input. Finally, 100% of users indicated a pref- erence for multimodal interaction over speech-only interaction with maps. These results indicate that for map-based tasks, users would both perform bet- ter and be more satisfied when using a multimodal interface. As an illustrative example, in the dis- tributed simulation application we describe in this paper, one user task is to add a "phase line" to a map. In the existing unimodal interface for this ap- plication (CommandTalk, Moore 1997), this is ac- complished with a spoken utterance such as 'CRE- ATE A LINE FROM COORDINATES NINE FOUR THREE NINE THREE ONE TO NINE EIGHT NINE NINE FIVE ZERO AND CALL IT PHASE …

Content truncated to 500 words