SemEval-2010 Task 12: Parser Evaluation Using Textual Entailments

Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 51–56, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics SemEval-2010 Task 12: Parser Evaluation using Textual Entailments Deniz Yuret Koc¸ University ˙Istanbul, Turkey dyuret@ku.edu.tr Aydın Han Koc¸ University ˙Istanbul, Turkey ahan@ku.edu.tr Zehra Turgut Koc¸ University ˙Istanbul, Turkey zturgut@ku.edu.tr Abstract Parser Evaluation using Textual Entail- ments (PETE) is a shared task in the SemEval-2010 Evaluation Exercises on Semantic Evaluation. The task involves recognizing textual entailments based on syntactic information alone. PETE intro- duces a new parser evaluation scheme that is formalism independent, less prone to annotation error, and focused on semanti- cally relevant distinctions. 1 Introduction Parser Evaluation using Textual Entailments (PETE) is a shared task that involves recognizing textual entailments based on syntactic information alone. Given two text fragments called “text” and “hypothesis”, textual entailment recognition is the task of determining whether the meaning of the hypothesis is entailed (can be inferred) from the text. In contrast with general RTE tasks (Dagan et al., 2009) the PETE task focuses on syntactic entailments: Text: The man with the hat was tired. Hypothesis-1: The man was tired. (yes) Hypothesis-2: The hat was tired. (no) PETE is an evaluation scheme based on a natu- ral human linguistic competence (i.e. the ability to comprehend sentences and answer simple yes/no questions about them). We believe systems should try to model natural human linguistic competence rather than their dubious competence in artificial tagging tasks. The PARSEVAL measures introduced nearly two decades ago (Black et al., 1991) still dominate the field of parser evaluation. These methods com- pare phrase-structure bracketings produced by the parser with bracketings in the annotated corpus, or “treebank”. Parser evaluation using short textual entailments has the following advantages com- pared to treebank based evaluation. Consistency: Recognizing syntactic entail- ments is a more natural task for people than treebank annotation. Focusing on a natural human competence makes it practical to collect high quality evaluation data from untrained annotators. The PETE dataset was annotated by untrained Amazon Mechanical Turk workers at an insignificant cost and each annotation is based on the unanimous agreement of at least three workers. In contrast, of the 36306 constituent strings that appear multiple times in the Penn Treebank (Marcus et al., 1994), 5646 (15%) have multiple conflicting annotations. If indicative of the general level of inconsistency, 15% is a very high number given that the state of the art parsers claim f-scores above 90% (Charniak and Johnson, 2005). Relevance: PETE automatically focuses atten- tion on semantically relevant phenomena rather than differences in annotation style or linguistic convention. Whether a phrase is tagged ADJP vs ADVP rarely affects semantic interpretation. At- taching the wrong subject to a verb or the wrong prepositional phrase to a noun changes the mean- ing of the sentence. Standard treebank based eval- uation metrics do not distinguish between seman- tically relevant and irrelevant errors (Bonnema et al., 1997). In PETE semantically relevant differ- ences lead to different entailments, semantically irrelevant differences do …

Content truncated to 500 words