SCHWA: PETE Using CCG Dependencies with the C&C Parser

Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 313–316, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics SCHWA: PETE using CCG Dependencies with the C&C Parser Dominick Ng, James W. D. Constable, Matthew Honnibal and James R. Curran e -lab, School of Information Technologies University of Sydney NSW 2006, Australia {dong7223,jcon6353,mhonn,james}@it.usyd.edu.au Abstract This paper describes the SCHWA system entered by the University of Sydney in Se- mEval 2010 Task 12 – Parser Evaluation using Textual Entailments (Yuret et al., 2010). Our system achieved an overall ac- curacy of 70% in the task evaluation. We used the C&C parser to build CCG de- pendency parses of the truth and hypothe- sis sentences. We then used partial match heuristics to determine whether the sys- tem should predict entailment. Heuristics were used because the dependencies gen- erated by the parser are construction spe- cific, making full compatibility unlikely. We also manually annotated the develop- ment set with CCG analyses, establishing an upper bound for our entailment system of 87%. 1 Introduction The SemEval 2010 Parser Evaluation using Tex- tual Entailments (PETE) task attempts to address the long-standing problems in parser evaluation caused by the diversity of syntactic formalisms and analyses in use. The task investigates the feasibility of a minimalist extrinsic evaluation – that of detecting textual entailment between a truth sentence and a hypothesis sentence. It is extrin- sic in the sense that it evaluates parsers on a task, rather than a direct comparison of their output against some gold standard. However, it requires only minimal task-specific logic, and the proposed entailments are designed to be inferrable based on syntactic information alone. Our system used the C&C parser (Clark and Curran, 2007a), which uses the Combinatory Cat- egorial Grammar formalism (CCG, Steedman, 2000). We used the CCGbank-style dependency output of the parser (Hockenmaier and Steedman, 2007), which is a directed graph of head-child re- lations labelled with the head’s lexical category and the argument slot filled by the child. We divided the dependency graphs of the truth and hypothesis sentences into predicates that con- sisted of a head word and its immediate children. For instance, the parser’s analysis of the sentence Totals include only vehicle sales reported in pe- riod might produce predicates like include(Totals, sales), only(include), and reported(sales). If at least one such predicate matches in the two parses, we predict entailment. We consider a single pred- icate match sufficient for entailment because the lexical categories and slots that constitute our de- pendency labels are often different in the hypothe- sis sentence due to the generation process used in the task. The single predicate heuristic gives us an over- all accuracy of 70% on the test set. Our precision and recall over the test set was 68% and 80% re- spectively giving an F-score of 74%. To investigate how many of the errors were due to parse failures, and how many were failures of our entailment recognition process, we manually annotated the 66 development truth …

Content truncated to 500 words