Designing a Task-Based Evaluation Methodology for a Spoken Machine Translation System

Designing a Task-Based Evaluation Methodology for a Spoken Machine Translation System Kavita Thomas Language Technologies Institute Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213, USA kavita@cs, cmu. edu Abstract In this paper, I discuss issues pertinent to the design of a task-based evaluation methodology for a spoken machine translation (MT) sys- tem processing human to human communica- tion rather than human to machine communi- cation. I claim that system mediated human to human communication requires new evaluation criteria and metrics based on goal complexity and the speaker's prioritization of goals. 1 Introduction Task-based evaluations for spoken language sys- tems focus on evaluating whether the speaker's task is achieved, rather than evaluating utter- ance translation accuracy or other aspects of system performance. Our MT project focuses on the travel reservation domain and facilitates on-line translation of speech between clients and travel agents arranging travel plans. Our prior evaluations (Gates et al., 1996) have focused on end-to-end translation accuracy at the ut- terance level (i.e., fraction of utterances trans- lated perfectly, acceptably, and unacceptably). While this method of evaluation conveys trans- lation accuracy, it does not give any information about how many of the client's travel arrange- ment goals have been conveyed, nor does it take into account the complexity of the speaker's goals and task, or the priority that they assign to their goals; for example, the same end-to-end score for two dialogues may hide the fact that in one dialogue the speakers were able to com- municate their most important goals while in the other they were only able to communicate successfully the less important goals. One common approach to evaluating spoken language systems focusing on human-machine dialogue is to compare system responses to cor- rect reference answers; however, as discussed by (Walker et al., 1997), the set of reference answers for any particular user query is tied to the system's dialogue strategy. Evaluation methods independent of dialogue strategy have focused on measuring the extent to which sys- tems for interactive problem solving aid users via log-file evaluations (Polifroni et al., 1992), quantifying repair attempts via turn correction ratio, tracking user detection and correction of system errors (Hirschman and Pao, 1993), and considering transaction success (Shriberg et al., 1992). (Danieli and Gerbino, 1995) measure the dialogue module's ability to recover from partial failures of recognition or understanding (i.e., implicit recovery) and inappropriate utter- ance ratio; (Simpson and Fraser, 1993) discuss applying turn correction ratio, transaction suc- cess, and contextual appropriateness to dialogue evaluations, and (Hirschman et ah, 1990) dis- cuss using task completion time as a black box evaluation metric. Current literature on task-based evaluation methodologies for spoken language systems pri- marily focuses on human-computer interactions rather than system-mediated human-human in- teractions. For a multilingual MT system, speakers communicate via the system, which translates their responses and generates the out- put in the target language via speech synthesis. Measuring solution quality (Sikorski and Allen, 1995), transaction success, or contextual appro- priateness is meaningless, since we are not in- terested …

Content truncated to 500 words