Development and Preliminary Evaluation of the MIT ATIS System

Development and Preliminary Evaluation of the MIT ATIS System 1 Stephanie Seneff, James Glass, David Goddeau, David Goodine, Lynette Hirschman, Hong Leung, Michael Phillips, Joseph Polifroni, and Victor Zue Spoken Language Systems Group Laboratory for Computer Science Massachusetts Institute of Technology Cambridge, Massachusetts 02139 ABSTRACT This paper represents a status report on the MIT ATIS sys- tem. The most significant new achievement is that we now have a speech-input mode. It is based on the MIT SUMMIT system us- ing context independent phone models, and includes a word-pair grammar with perplexity 92 (on the June-90 test set). In addi- tion, we have completely redesigned the back-end component, in order to emphasize portability and extensibility. The parser now produces an intermediate semantic frame representation, which serves as the focal point for all back-end operations, such as his- tory management, text generation, and SQL query generation. Most of those aspects of the system that are tied to a partic- ular domain are now entered through a set of tables associated with a small artificial language for decoding them. We have also improved the display of the database table, making it consider- ably easier for a subject to comprehend the information given. We report here on the results of the official DARPA February-91 evaluation, as well as on results of an evaluation on data collected at MIT, for both speech input and text input. INTRODUCTION In June 1990, we reported on the initial development of the MIT ATIS system, and participated in the first round of DARPA common evaluation using text input [5]. Since then, a number of changes have been made to our system, particu- larly the back-end component that transforms the parse tree into a representation that can be used to maintain discourse, generate confirmation messages, and produce SQL queries for accessing the OAG database. We have also connected the SUMMIT speech recognizer to our ATIS system, so that it can now accept verbal input. This paper gives a progress report on the MIT ATIS de- velopment, with particular emphasis on how the system pro- cesses an input utterance to achieve understanding and gen- erate responses. We will also report on the evaluation of the system for both text and speech input, using the data provided by TI through NIST, as well as the data that we XThis research was supported by DARPA under Contract N00014- 89-J-1332, monitored through the Office of Naval Research. have collected over the past few months [3]. Aspects of the system involving discourse and dialogue are based on simi- lar principles as before, but has been modified to reflect the new semantic representations. A detailed description of our discourse model can be found in a companion paper [4]. SYSTEM DESCRIPTION In this section we will describe those aspects of the system that have changed significantly since our report last June [5]. The most significant change has been the incorporation of the speech recognition component. We begin by describing the recognizer configuration and the interface mechanism we …

Content truncated to 500 words