Semantic Evaluation for Spoken-Language Systems

SEMANTIC EVALUATION FOR SPOKEN-LANGUAGE SYSTEMS Robert C. Moore Artificial Intelligence Center SRI International Menlo Park, California 94025 ABSTRACT Development has begun on a semantic evaluation (Se- mEval) methodology and infrastructure for the ARPA Spo- ken Language Program. SemEval is an attempt to define a task-independent technology-based evaluation for language- understanding systems consisting of three parts: word-sense identification, predicate-argument structure determination, and identification of coreference relations. An initial spoken- language SemEval on ATIS data is planned for Novem- ber/December 1994, concurrent with the next ATIS CAS (database answer) evaluation. 1. INTRODUCTION Since the summer of 1993, there has been considerable discus- sion in the ARPA HLT community of moving the evaluation of understanding systems for both spoken and written lan- guage away from application-based metrics (such as correct database response in ATIS, or template fills in MUG) towaxd technology-based metrics. The benefits hoped to be derived from such a shift include greater focus on underlying tech- nology issues, rather than application issues, and lowering the overhead required to participate in evaluations in terms of developing application systems. The discussions have fo- cused on the concept of a semantic evaluation, or "SemEval" consisting of three components: word-sense identification, predicate-argument structure determination, and coreference determination. This paper reports on how these ideas are being developed within the ARPA spoken-language commu- nity, in preparation for an initial spoken-language SemEval on ATIS data concurrent with the ATIS GAS (database an- swer) evaluation planned for November/December 1994. A meeting was held at SRI, 21-23 October 1993, to begin fleshing out these ideas for the evaluation of spoken-language understanding systems. The meeting was attended by re- searchers, annotators, and evaluators involved in both the ARPA Spoken Language Program and the ARPA Written Language Program: Fernando Pereira from AT&T Bell Labo- ratories; Rusty Bobrow and Dave Stallaxd from BBN; Wayne Ward and Sergei Nirenburg from CMU; Stephanie Seneff and Eric Brill from MIT; Robert Moore, Kate Hunicke-Smith, Jerry Hobbs, Harry Bratt, and Mark Gawron from SRI; Deb- hie Dald and Lew Norton from Unisys; Mitch Marcus and Grace Kim from the University of Pennsylvania; Nancy Chin- chor from SAIG; George Doddingtou from ARPA; George Miller from Princeton University; Dave Pallett and Bruce Lurid from NIST; and Ralph Grishman from NYU. This pa- per is derived from the discussions at the October meeting and from subsequent proposals made and discussed by the participants. 2. WHY DO SEMEVAL? A question that has received, and continues to receive, exten- sive discussion is ~'Why do we want to do SemEval at allT" George Doddington [1] has addressed this from ARPA's per- spective as follows: Why is SemEval a good idea? Well, first, ARPA's goal in the HLT Program is to make strate- gic advances in core human language technology. This goal is a technology goal (to produce useful functionality). It is not a science goal (to produce scientific understanding), nor is it an application goal (to produce useful applications embodying hu- man language technology). So, why do SemEval instead of only doing …

Content truncated to 500 words