Semantics-based Representation for Multimodal Interpretation in Conversational Systems

Abstract To support context-based multimodal interpre- tation in conversational systems, we have devel- oped a semantics-based representation capture salient information from user inputs andto the overall conversation. In particular, we present three unique characteristics: fine- grained semantic models, flexible composition of feature structures, and consistent representa- tion at multiple levels. This representation allows our system to use rich contexts to resolve ambiguities, infer unspecified information, and improve multimodal alignment. As a result, our system is able to enhance understanding of mul- timodal inputs including those abbreviated, imprecise, or complex ones. 1 Introduction Inspired by earlier works on multimodal interfaces (e.g., Bolt, 1980; Cohen el al., 1996; Wahlster, 1991; Zancanaro et al., 1997), we are currently building an intelligent infrastructure, called Responsive Informa- tion Architect (RIA) to aid users in their informa- tion-seeking process. Specifically, RIA engages users in a full-fledged multimodal conversation, where users can interact with RIA through multiple modalities (speech, text, and gesture), and RIA can act/react through automated multimedia generation (speech and graphics) (Zhou and Pan 2001). Cur- rently, RIA is embodied in a testbed, called Real HunterTM, a real-estate application to help users find residential properties. As a part of this effort, we are building a seman- tics-based multimodal interpretation framework MIND (Multimodal Interpretation for Natural Dia- log) to identify meanings of user multimodal inputs. Traditional multimodal interpretation has been focused on integrating multimodal inputs together with limited consideration on the interaction context. In a conversation setting, user inputs could be abbre- viated or imprecise. Only by combining multiple inputs together often cannot reach a full understand- ing. Therefore, MIND applies rich contexts (e.g., conversation context and domain context) enhance multimodal interpretation. In support of thisto context-based approach, we have designed a seman- tics-based representation to capture salient informa- tion from user inputs and the overall conversation. In this paper, we will first give a brief overview on multimodal interpretation in MIND. Then we will present our semantics-based representation and dis- cuss its characteristics. Finally, we will describe the use of this representation in context-based multimo- dal interpretation and demonstrate that, with this rep- resentation, MIND is able to process a variety of user inputs including those ambiguous, abbreviated and complex ones. 2 Multimodal Interpretation To interpret user multimodal inputs, MIND takes three major processes as in Figure 1: unimodal understanding, multimodal understanding, and dis- course understanding. During unimodal understand- ing, MIND applies modality specific recognition and understanding components (e.g., a speech recognizer and a language interpreter) to identify meanings from each unimodal input, and captures those mean- ings in a representation called modality unit. During multimodal understanding, MIND combines seman- tic meanings of unimodal inputs (i.e., modality units), and uses contexts (e.g., conversation context and domain context) to form an overall understand- ing of user multimodal inputs. Such an overall understanding is then captured in a representation called conversation unit. Furthermore, MIND also identifies how an input relates to the overall conver- sation discourse through discourse understanding. In particular, MIND uses a representation called con- …

Content truncated to 500 words