SemEval-2007 Task 19: Frame Semantic Structure Extraction

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 99–104, Prague, June 2007. c⃝2007 Association for Computational Linguistics SemEval’07 Task 19: Frame Semantic Structure Extraction Collin Baker, Michael Ellsworth International Computer Science Institute Berkeley, California {collinb,infinity} @icsi.berkeley.edu Katrin Erk Computer Science Dept. University of Texas Austin katrin.erk@mail.utexas.edu Abstract This task consists of recognizing words and phrases that evoke semantic frames as defined in the FrameNet project (http: //framenet.icsi.berkeley.edu), and their semantic dependents, which are usually, but not always, their syntactic dependents (including subjects). The train- ing data was FN annotated sentences. testing, participants automatically annotatedIn three previously unseen texts to match gold standard (human) annotation, including pre- dicting previously unseen frames and roles. Precision and recall were measured both for matching of labels of frames and FEs and for matching of semantic dependency trees based on the annotation. 1 Introduction The task of labeling frame-evoking words with ap- propriate frames is similar to WSD, while the task of assigning frame elements is called Semantic Role Labeling (SRL), and has been the subject of several shared tasks at ACL and CoNLL. For example, in the sentence “Matilde said, ‘I rarely eat rutabaga,”’ said evokes the Statement frame, and eat evokes the Ingestion frame. The role of SPEAKER in the Statement frame is filled by Matilda, and the role of MESSAGE, by the whole quotation. In the Inges- tion frame, I is the INGESTOR and rutabaga fills the INGESTIBLES role. Since the ingestion event is con- tained within the MESSAGE of the Statement event, we can represent the fact that the message conveyed was about ingestion, just by annotating the sentence with respect to these two frames. After training on FN annotations, the participants’ systems labeled three new texts automatically. The evaluation measured precision and recall for frames and frame elements, with partial credit for incorrect but closely related frames. Two types of evaluation were carried out: Label matching evaluation, in which the participant’s labeled data was compared directly with the gold standard labeled data, and Se- mantic dependency evaluation, in which both the gold standard and the submitted data were first con- verted to semantic dependency graphs in XML for- mat, and then these graphs were compared. There are three points that make this task harder and more interesting than earlier SRL tasks: (1) while previous tasks focused on role assignment, the current task also comprises the identification of the appropriate FrameNet frame, similar to WSD, (2) the task comprises not only the labeling of individ- ual predicates and their arguments, but also the inte- gration of all labels into an overall semantic depen- dency graph, a partial semantic representation of the overall sentence meaning based on frames and roles, and (3) the test data includes occurrences of frames that are not seen in the training data. For these cases, participant systems have to identify the closest known frame. This is a very realistic sce- nario, encouraging the development of robust sys- tems showing graceful degradation in the face …

Content truncated to 500 words