Finding Errors Automatically in Semantically Tagged Dialogues

Finding Errors Automatically in Semantically Tagged Dialogues John Aberdeen, Christine Doran, Laurie Damianos, Samuel Bayer and Lynette Hirschman The MITRE Corporation 202 Burlington Road Bedford, MA 01730 USA +1.781.271.2000 {aberdeen,cdoran,laurie,sam,lynette}@mitre.org ABSTRACT We describe a novel method for detecting errors in task-based human-computer (HC) dialogues by automatically deriving them from semantic tags. We examined 27 HC dialogues from the DARPA Communicator air travel domain, comparing user inputs to system responses to look for slot value discrepancies, both automatically and manually. For the automatic method, we labeled the dialogues with semantic tags corresponding to "slots" that would be filled in "frames" in the course of the travel task. We then applied an automatic algorithm to detect errors in the dialogues. The same dialogues were also manually tagged (by a different annotator) to label errors directly. An analysis of the results of the two tagging methods indicates that it may be possible to detect errors automatically in this way, but our method needs further work to reduce the number of false errors detected. Finally, we present a discussion of the differing results from the two tagging methods. Keywords Dialogue, Error detection, DARPA Communicator. 1. INTRODUCTION In studying the contrasts between human-computer (HC) and human-human (HH) dialogues [1] it is clear that many HC dialogues are plagued by disruptive errors that are rarely seen in HH dialogues. A comparison of HC and HH dialogues may help us understand such errors. Conversely, the ability to detect errors in dialogues is critical to understanding the differences between HC and HH communication. Understanding HC errors is also crucial to improving HC interaction, making it more robust, trustworthy and efficient. The goal of the work described in this paper is to provide an annotation scheme that allows automatic calculation of misunderstandings and repairs, based on semantic information presented at each turn. If we represent a dialogue as a sequence of pairs of partially-filled semantic frames (one for the user’s utterances, and one for the user’s view of the system state), we can annotate the accumulation and revision of information in the paired frames. We hypothesized that, with such a representation, it would be straightforward to detect when the two views of the dialogue differ (a misunderstanding), where the difference originated (source of error), and when the two views reconverge (correction). This would be beneficial because semantic annotation often is used for independent rea- sons, such as measurements of concepts per turn [8], information bit rate [9], and currently active concepts [10]. Given this, if our hypothesis is correct, then by viewing semantic annotation as a representation of filling slots in user and system frames, it should be possible to detect errors automatically with little or no additional annotation. 2. SEMANTIC TAGGING We tagged 27 dialogues from 4 different systems that participated in a data collection conducted by the DARPA Communicator program in the summer of 2000. These are dialogues between paid subjects and spoken language dialogue systems operating in the air travel domain. Each dialogue was labeled with …

Content truncated to 500 words