Experiments in Evaluating Interactive Spoken Language Systems

Experiments in Evaluating Interactive Spoken Language Systems 1 Joseph Polifroni, Lynette Hirschman, Stephanie Seneff, and Victor Zue Spoken Language Systems Group Laboratory for Computer Science Massachusetts Institute of Technology Cambridge, Massachusetts 02139 ABSTRACT As the DARPA spoken language community moves to- wards developing useful systems for interactive problem solv- ing, we must explore alternative evaluation procedures that measure whether these systems aid people in solving problems within the task domain. In this paper, we describe several experiments exploring new evaluation procedures. To look at end-to-end evaluation, we modified our data collection pro- cedure slightly in order to experiment with several objective task completion measures. We found that the task completion time is well correlated with the number of queries used. We also explored log file evaluation, where evaluators were asked to judge the clarity of the query and the correctness of the re- sponse based on examination of the log file. Our results show that seven evaluators were unanimous on more than 80% of the queries, and that at least 6 out of 7 evaluators agreed over 90% of the time. Finally, we applied these new procedures to compare two systems, one system requiring a complete parse and the other using the more flexible robust parsing mecha- nism. We found that these metrics could distinguish between these systems: there were significant differences in ability to complete the task, number of queries required to complete the task, and score (as computed through a log file evaluation) between the robust and the non-robust modes. INTRODUCTION For the first two years of the DARPA Spoken Lan- guage Program, common evaluation in the ATIS domain has been performed solely with the Common Answer Specification (CAS) protocol [4], whereby a system's per- formance is determined by comparing its output, ex- pressed as a set of database tuples, with one or more predetermined reference answers [1]. The CAS protocol has the advantage that system evaluation can be car- ried out automatically, once the principles for generating the reference answers have been established and a cor- pus has been annotated accordingly. Since direct com- parison across systems can be performed relatively easily with this procedure, we have been able to achieve cross fertilization of research ideas, leading to rapid research progress. 1This research was supported by DARPA under Contract N000] 4-89-J-1332, monitored through the Office of Naval Research. 28 QUERY 1: RESPONSE I: QUERY 2: RESPONSE 2: PLEASE LIST THE FLIGHT FROM PITTSBURGH TO BALTIMORE THAT WILL HE MADE BY SIX SEAT AIRPLANE ON JUNE TWENTIETH There are no flights from pittsburgh to Baltimore leaving before 6:00 a.m. on Saturday June 20. LIST THE NUMBER OF FIRST CLASS FLIGHTS AVAILABLE ON DELTA AIRLINES 279 Table 1: Two examples of queries from the February '92 Evaluation for which the CAS evaluation gives a misleading assessment of the system's ability to understand. However, CAS evaluation is not without its share of shortcomings. One unfortunate outcome of evaluat- ing systems based on answers retrieved from the flight database is that …

Content truncated to 500 words