372:Comparing the Benefit of Different Dependency Parsers for Textual Entailment Using Syntactic Constraints Only

Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 308–312, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics 372:Comparing the Benefit of Different Dependency Parsers for Textu- al Entailment Using Syntactic Constraints Only Alexander Volokh Günter Neumann alexander.volokh@dfki.de neumann@dfki.de DFKI DFKI Stuhlsatzenhausweg 3 Stuhlsatzenhausweg 3 66123 Saarbrücken, Germany 66123 Saarbrücken, Germany Abstract We compare several state of the art dependency parsers with our own parser based on a linear classification technique. Our primary goal is therefore to use syntactic information only, in or- der to keep the comparison of the parsers as fair as possible. We demonstrate, that despite the in- ferior result using the standard evaluation metrics for parsers like UAS or LAS on standard test data, our system achieves comparable results when used in an application, such as the SemEv- al-2 #12 evaluation exercise PETE. Our submis- sion achieved the 4th position out of 19 participat- ing systems. However, since it only uses a linear classifier it works 17-20 times faster than other state of the parsers, as for instance MaltParser or Stanford Parser. 1 Introduction Parsing is the process of mapping sentences to their syntactic representations. These representa- tions can be used by computers for performing many interesting natural language processing tasks, such as question answering or information extraction. In recent years a lot of parsers have been developed for this purpose. A very interesting and important issue is the comparison between a large number of such pars- ing systems. The most widespread method is to evaluate the number of correctly recognized units according to a certain gold standard. For depend- ency-based units unlabeled or labeled attachment scores (percentage of correctly classified depend- ency relations, either with or without the depend- ency relation type) are usually used (cf. Buchholz and Marsi, 2006). However, parsing is very rarely a goal in itself. In most cases it is a necessary preprocessing step for a certain application. Therefore it is usually not the best option to decide which parser suits one's goals best by purely looking on its perform- ance on some standard test data set. It is rather more sensible to analyse whether the parser is able to recognise those syntactic units or rela- tions, which are most relevant for one's applica- tion. The shared task #12 PETE in the SemEval- 2010 Evaluation Exercises on Semantic Evalu- ation (Yuret, Han and Turgut, 2010) involved re- cognizing textual entailments (RTE). RTE is a binary classification task, whose goal is to de- termine, whether for a pair of texts T and H the meaning of H is contained in T (Dagan et al., 2006). This task can be very complex depending on the properties of these texts. However, for the data, released by the organisers of PETE, only the syntactic information should be sufficient to reli- ably perform this task. Thus it offers an ideal set- ting for evaluating the performance of different parsers. To our mind evaluation of parsers via RTE is a very good …

Content truncated to 500 words