Multi-Human Dialogue Understanding for Assisting Artifact-Producing Meetings

Multi-Human Dialogue Understanding for Assisting Artifact-Producing Meetings John Niekrasz and Alexander Gruenstein and Lawrence Cavedon Center for the Study of Language and Information (CSLI) Stanford University Cordura Hall, Stanford, CA, 94305-4115, USA http://www-csli.stanford.edu/semlab/ {niekrasz, alexgru, lcavedon}@csli.stanford.edu Abstract In this paper we present the dialogue- understanding components of an architec- ture for assisting multi-human conversa- tions in artifact-producing meetings: meet- ings in which tangible products such as project planning charts are created. Novel aspects of our system include multimodal ambiguity resolution, modular ontology- driven artifact manipulation, and a meeting browser for use during and after meetings. We describe the software architecture and demonstrate the system using an example multimodal dialogue. 1 Introduction Recently, much attention has been focused on the domain of multi-person meeting under- standing. Meeting dialogue presents a wide range of challenges including continuous multi- speaker automatic speech recognition (ASR), 2D whiteboard gesture and handwriting recog- nition, 3D body and eye tracking, and multi- modal multi-human dialogue management and understanding. A significant amount of re- search has gone toward understanding the prob- lems facing the collection, organization, and visualization of meeting data (Moore, 2002; Waibel et al., 2001), and meeting corpora like the ICSI Meeting Corpus (Janin et al., 2003) are being made available. Continuing research in the multimodal meeting domain has since blos- somed, including ongoing work from projects such as AMI1 and M42, and efforts from sev- eral institutions. Previous work on automatic meeting un- derstanding has mostly focused on surface- level recognition, such as speech segmentation, for obvious reasons: understanding free multi- human speech at any level is an extremely diffi- cult problem for which best performance is cur- rently poor. In addition, the primary focus for 1http://www.amiproject.org/ 2http://www.m4project.org/ applications has been on off-line tools such as post-meeting multimodal information browsing. In parallel to such efforts we are applying dialogue-management techniques to attempt to understand and monitor meeting dialogues as they occur, and to supplement multimodal meeting records with information relating to the structure and purpose of the meeting. Our efforts are focused on assisting artifact- producing meetings, i.e. meetings for which the intended outcome is a tangible product such as a project management plan or a budget. The dialogue-understanding system helps to create and manipulate the artifact, delivering a final product at the end of the meeting, while the state of the artifact is used as part of the dia- logue context under which interpretation of fu- ture utterances is performed, serving a num- ber of useful roles in the dialogue-understanding process: • The dialogue manager employs generic di- alogue moves with plugin points to be de- fined by specific artifact types, e.g. project plan, budget; • The artifact state helps resolve ambiguity by providing evidence for multimodal fu- sion and constraining topic-recognition; • The artifact type can be used to bias ASR language-models; • The constructed artifact provides a inter- face for a meeting browser that supports directed queries about discussion that took place in the meeting, e.g. “Why did we decide on that …

Content truncated to 500 words