SRI International Results February 1992 ATIS Benchmark Test

SRI INTERNATIONAL RESULTS FEBRUARY 1992 ATIS BENCHMARK TEST Douglas E. Appelt, Eric Jackson SRI International Menlo Park, CA 94025 ABSTRACT We describe the results that SRI International achieved on the February 1992 ATIS Speech and Natural Language Sys- tem Test. The basic architecture of the system is described, including a set of parameters capable of altering the system's behavior and processing strategy. We report on several ex- periments that were run on the February test set to evaluate several processing strategies for both natural-language only and full spoken-language system tests. 1. INTRODUCTION This paper reports on the results of running SRI In- ternational's spoken-language system on the DARPA- sponsored February 1992 test. The system's natural- language processing has been parameterized in several ways to achieve different behaviors. In addition to run- ning our system with what we believed at the time of the test to be the optimal parameter settings to produce our official results, we have conducted some experiments by running the system with a variety of parameter settings. The results of these experiments shed some light on the trade-offs among various SLS and natural-language pro- cessing strategies, and provide some interesting data for evaluating the evaluation methodology itself. 2. SYSTEM DESCRIPTION The SLS system used for the February evaluation is an integration of the SRI DECIPHER speech recognition [1,4,5] system with the SRI TRAVELOGUE natural- language processing system. The integration between these two systems is currently accomplished by a sim- ple serial interface: the best accoustic hypothesis is pro- cessed by the NL system to produce the answer to the query. The DECIPHER System DECIPHER is a speaker-independent continuous-speech speech recognition system based on tied-mixture Hid- den Markov Model (HMM) models. It uses six features, three being vectors (cepstra, delta-cepstra, and delta- delta-cepstra) and three scalars (energy, delta-energy, and delta-delta-energy). These features are computed 95 from a filter bank that is derived via an FFT and high- pass filtered (RASTA filtered) in the log-spectral-energy domain. DECIPHER models pronunciation variability through word networks generated by linguistic rules then pruned probabilistically. There are cross-word acoustic and phonological models. Parallel recognizers were im- plemented and trained separately on male and female speech. The DECIPHER-ATIS system uses a backed-off bigram language model to reduce the perplexity of the input speech. The acoustic models were trained on all available ATIS spontaneous and read data (excluding 809 sentences used for system development that include 362 October 1991 dry run sentences and 447 MADCOW sentences). The backed-off bigram language model was trained on the available ATIS spontaneous speech data. This included 14,779 sentences (approximately 150,000 words). The recognition lexicon consisted of all words spoken in all available spontaneous ATIS data. There are also lexi- cal entries for breaths and silence. No catch-all rejection model was used for out-of-vocabulary items. The vocab- ulary size is 1385 words. The TRAVELOGUE System The TRAVELOGUE system consists of a template- matching sentence-analysis mechanism [3] coupled with a context-handling mechanism and a database query generation component. The template matcher operates …

Content truncated to 500 words