A Three-Tiered Evaluation Approach for Interactive Spoken Dialogue Systems

A Three-Tiered Evaluation Approach for Interactive Spoken Dialogue Systems Kathleen Stibler and James Denny Lockheed Martin Advanced Technology Laboratories 1 Federal Street, A&E 3W Camden NJ 08102 {kcomegno, jdenny}@atl.lmco.com ABSTRACT We describe a three-tiered approach for evaluation of spoken dialogue systems. The three tiers measure user satisfaction, system support of mission success and component performance. We describe our use of this approach in numerous fielded user studies conducted with the U.S. military. Keywords Evaluation, spoken language system, spoken dialogue system 1. INTRODUCTION Evaluation of spoken language systems is complicated by the need to balance distinct goals. For collaboration with others in the speech technology community, metrics must be generic enough for comparison to analogous systems. For project man- agement and business purposes, metrics must be specific enough to demonstrate end-user utility and improvement over other approaches to a problem. Since 1998, we have developed a spoken language dialogue technology called Listen-Communicate-Show (LCS) and applied it to demonstration systems for U.S. Marines logistics, U.S. Army test data collection, and commercial travel reserva- tions. Our focus is the transition of spoken dialogue technol- ogy to military operations. We support military users in a wide range of tasks under diverse conditions. Therefore, our defini- tion of success for LCS is operational success. It must reflect the real world success of our military users in performing their tasks. In addition, for our systems to be considered successful, they must be widely usable and easy for all users to operate with minimal training. Our evaluation methodology must model these objectives. With these goals in mind, we have developed a three-tier metric system for evaluating spoken language system effective- ness. The three tiers measure (1) user satisfaction, (2) system support of mission success and (3) component performance. 2. THE THREE-TIERED APPROACH Our three-tier metric scheme evaluates multiple aspects of LCS system effectiveness. User satisfaction is a set of subjective measures that introduces user perceptions into the assessment of the system. System support of mission success measures overall system performance with respect to our definition of success. Component performance scores the individual sys- tem componentÕs role in overall system success. Collection of user input is essential in evaluation for two rea- sons. First, it is necessary to consider user perspective during evaluation to achieve a better understanding of user needs. Second, user preference can influence interpretation of success measurements of mission success and component performance. Mission success and component performance are often tradeoffs, with inefficient systems producing higher scores of success. Since some users are willing to overlook efficiency for guaran- teed performance while others opt for efficiency, our collection of user input helps determine the relative importance of these aspects. Mission success is difficult to quantify because it is defined differently by users with different needs. Therefore, it is essen- tial to establish a definition of mission success early in the evaluation process. For our applications, we derive this defini- tion from domain knowledge acquisition with potential users. It is important to evaluate components individually since …

Content truncated to 500 words