Cambridge: Parser Evaluation Using Textual Entailment by Grammatical Relation Comparison

Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 268–271, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics Cambridge: Parser Evaluation using Textual Entailment by Grammatical Relation Comparison Laura Rimell and Stephen Clark University of Cambridge Computer Laboratory {laura.rimell,stephen.clark}@cl.cam.ac.uk Abstract This paper describes the Cambridge sub- mission to the SemEval-2010 Parser Eval- uation using Textual Entailment (PETE) task. We used a simple definition of en- tailment, parsing both T and H with the C&C parser and checking whether the core grammatical relations (subject and object) produced for H were a subset of those for T. This simple system achieved the top score for the task out of those systems sub- mitted. We analyze the errors made by the system and the potential role of the task in parser evaluation. 1 Introduction SemEval-2010 Task 12, Parser Evaluation using Textual Entailment (PETE) (Yuret et al., 2010), was designed as a new, formalism-independent type of parser evaluation scheme. The task is broadly Recognizing Textual Entailment (RTE), but unlike typical RTE tasks, its intention is to fo- cus on purely syntactic entailments, assuming no background knowledge or reasoning ability. For example, given a text (T) The man with the hat was tired., the hypothesis (H) The man was tired. is entailed, but The hat was tired. is not. A cor- rect decision on whether H is entailed can be used as a diagnostic for the parser’s analysis of (some aspect of) T. By requiring only a binary decision on the entailment, instead of a full syntactic anal- ysis, a parser can be evaluated while its underlying formalism remains a “black box”. Our system had two components: a parser, and an entailment system which decided whether T en- tails H based on the parser’s output. We distin- guish two types of evaluation. Task evaluation, i.e. the official task scoring, indicates whether the entailment decisions – made by the parser and en- tailment system together – tally with the gold stan- dard dataset. Entailment system evaluation, on the other hand, indicates whether the entailment sys- tem is an appropriate parser evaluation tool. In the PETE task the parser is not evaluated directly on the dataset, since the entailment system acts as intermediary. Therefore, for PETE to be a vi- able parser evaluation scheme, each parser must be coupled with an entailment system which accu- rately reflects the parser’s analysis of the data. 2 System We used the C&C parser (Clark and Curran, 2007), which can produce output in the form of grammatical relations (GRs), i.e. labelled head- dependencies. For example, (nsubj tired man) for the example in Section 1 represents the fact that the NP headed by man is the subject of the predicate headed by tired. We chose to use the Stanford Dependency GR scheme (de Marneffe et al., 2006), but the same approach should work for other schemes (and other parsers producing GRs). Our entailment system was very simple, and based on the assumption that H is a simplified …

Content truncated to 500 words