Evaluation Results for the Talk’n’Travel System

Evaluation Results for the Talk’n’Travel System David Stallard BBN Technologies, Verizon 70 Fawcett. St. Cambridge, MA, 02140 Stallard@bbn.com ABSTRACT We describe and present evaluation results for Talk’n’Travel, a spoken dialogue language system for making air travel plans over the telephone. Talk’n’Travel is a fully conversational, mixed- initiative system that allows the user to specify the constraints on his travel plan in arbitrary order, ask questions, etc., in general spoken English. The system was independently evaluated as part of the DARPA Communicator program and achieved a high success rate. . 1. INTRODUCTION This paper describes and presents evaluation results for Talk’n’Travel, a spoken language dialogue system for making complex air travel plans over the telephone. Talk’n’Travel is a research prototype system sponsored under the DARPA Communicator program (MITRE, 1999). Some other systems in the program are Ward and Pellom (1999), Seneff and Polifroni (2000) and Rudnicky et al (1999). The common task of this program is a mixed-initiative dialogue over the telephone, in which the user plans a multi-city trip by air, including all flights, hotels, and rental cars, all in conversational English over the telephone. A similar research program is the European ARISE project (Den Os et al, 1999). An earlier version of Talk’n’Travel was presented in (Stallard, 2000). The present paper presents and discusses results of an independent evaluation of Talk’n’Travel, recently conducted as part of the DARPA Communicator program. The next section gives a brief overview of the system. 2. SYSTEM OVERVIEW The figure shows a block diagram of Talk’n’Travel. Spoken language understanding is provided by statistical N-gram speech recognition and a robust language understanding component. A plan-based dialogue manager coordinates interaction with the user, handling unexpected user input more flexibly than conventional finite-state dialogue control networks. It works in tandem with a state management component that adjusts the current model of user intention based on the user’s last utterance in context. Meaning and task state are represented by the path constraint representation (Stallard, 2000). An inference component is included which allows the system to deduce implicit requirements from explicit statements by the user, and to retract them if the premises change. The system is interfaced to the Yahoo/Travelocity flight schedule website, for access to live flight schedule information. Queries to the website are spawned off in a separate thread, which the dialogue manager monitors ands reports on to the user. 3. DIALOGUE STRATEGY Talk’n’Travel employs both open-ended and directed prompts. Sessions begin with open prompts like "What trip would you to take?". The system then goes to directed prompts to get any information the user did not provide ("What day are you leaving?", etc). The user may give arbitrary information at any prompt, however. The system provides implicit confirmation of the change in task state caused by the user’s last utterance ("Flying from Boston to Denver tomorrow ") to ensure mutual understanding. The system seeks explicit confirmation in certain cases, for example where the user appears to be making a change in date of travel. Once sufficient …

Content truncated to 500 words