Senseval-3 task: Automatic labeling of semantic roles

SENSEVAL-3 TASK Automatic Labeling of Semantic Roles Kenneth C. Litkowski CL Research 9208 Gue Road Damascus, MD 20872 ken@clres.com Abstract The SENSEVAL-3 task to perform automatic labeling of semantic roles was designed to encourage research into and use of the FrameNet dataset. The task was based on the considerable expansion of the FrameNet data since the baseline study of automatic labeling of semantic roles by Gildea and Jurafsky. The FrameNet data provide an extensive body of “gold standard” data that can be used in lexical semantics research, as the basis for its further exploitation in NLP applications. Eight teams participated in the task, with a total of 20 runs. Discussions among participants during development of the task and the scoring of their runs contributed to a successful task. Participants used a wide variety of techniques, investigating many aspects of the FrameNet data. They achieved results showing considerable improvements from Gildea and Jurafsky’s baseline study. Importantly, their efforts have contributed considerably to making the complex FrameNet dataset more accessible. They have amply demonstrated that FrameNet is a substantial lexical resource that will permit extensive further research and exploitation in NLP applications in the future. Introduction Word-sense disambiguation has frequently been criticized as a task in search of a reason. Since a considerable portion of a sense inventory has only a single sense, the question has been raised whether the amount of effort required by disambiguation is worthwhile. Heretofore, the focus of disambiguation has been on the sense inventory and has not examined the major reason why we would have lexical knowledge bases: how the meanings would be represented and thus, available for use in natural language processing applications. At the present time, a major paradigm for representing meaning has emerged in frame semantics, specifically in the FrameNet project. A worthy objective for the Senseval community is the development of a wide range of methods for automating frame semantics, specifically identifying and labeling semantic roles in sentences. An important baseline study of this process has recently appeared in the literature (Gildea and Jurafsky, 2002). The FrameNet project (Johnson et al., 2003) has put together a body of hand-labeled data and the Gildea and Jurafsky study has put together a set of suitable metrics for evaluating the performance of an automatic system. 1 The Senseval-3 Task This Senseval-3 task calls for the development of systems to meet the same objectives as the Gildea and Jurafsky study. The data for this task is a sample of the FrameNet hand-annotated data. Evaluation of systems is measured using precision and recall of frame elements and overlap of a system’s frame element sentence positions with those identified in the FrameNet data. The basic task for Senseval-3 is: Given a sentence, a target word and its frame, identify the frame elements within that sentence and tag them with the appropriate frame element name. The FrameNet project has just released a major revision (FrameNet 1.1) to its database, with 487 frames using 696 distinctly-named frame elements (although it is …

Content truncated to 500 words