Recent Improvements and Benchmark Results for Paramax ATIS System

Recent Improvements and Benchmark Results for Paramax ATIS System Lewis M. Norton, Deborah A. Dahl, and Marcia C. Linebarger * Paramax Systems Corporation, a Unisys Company P.O. Box 517 Paoli, PA, 19301 the ABSTRACT This paper describes three relatively domain-independent ca- pabilities recently added to the Paramax spoken language un- derstanding system: non-monotonic reasoning, implicit refer- ence resolution, and database query paraphrase. In addition, we discuss the results of the February 1992 ATIS benchmark tests. We describe a variation on the standard evaluation metric which provides a more tightly controlled measure of progress. Finally, we briefly describe an experiment which we have done in extending the n-best speech/language inte- gration architecture to improving OCR accuracy. 1. INTRODUCTION In recent work on the Paramax spoken language under- standing system we have focused on domain-independent capabilities whose relevance extends beyond the ATIS ap- plication. These include non-monotonic reasoning, im- plicit reference resolution, and the ability to create a simple paraphrase of a query for feedback to the user. Although these capabilities were motivated by ATIS data, they are likely to be required for processing data from other domains. We describe these improvements in the following section. In addition, we discuss our results on the ATIS benchmark tests. Because test data vary from test to test, it can be difficult to know whether fluctua- tions in performance are real, or are due to idiosyncracies in the particular test data selected. We have experi- mented with an evaluation paradigm in which systems used in earlier tests run the test data from the current test, thus controlling for the effect of variations in the test data. We suggest that this paradigm could provide a valuable supplement to the official tests. 2. SYSTEM IMPROVEMENTS 2.1. Non-Monotonic Reasoning We previously described [1] a feature of the PUN- DIT natural language processing system whereby the *This paper was supported by DARPA contract N000014-89- C0171, administered by the Office of Naval Research, and by inter- nal funding from Paramax Systems Corporation (formerly Unisys Defense Systems). We wish to thank Suzanne Taylor for help- ful discussions on applying natural language constraints to OCR enhancement. system makes inferences involving more than one decomposition. 1 For example, the instantiated decom- positions produced for "flights leaving Boston" are: flight_C(flightl, source(_) .... ) leaveP(leavel, flight(flightl), source(boston) .... ) Application of a rule relating the leaveP and the flight_C decompositions results in the source slot of the flight_C decomposition being instantiated to "boston". We have extended this feature to make it possible to re- tract such inferences. This extension allows PUNDIT to do non-monotonic reasoning ([2]). That is, the system can make and reason from tentative inferences, which can be removed from the context if they are not sup- ported by the developing dialog. The facility has been implemented in a fully general way, so that any test that can be coded in PROLOG can be the trigger for retraction. Currently we use this capability to retract certain infer- ences which result in a database …

Content truncated to 500 words