MARS: A Specialized RTE System for Parser Evaluation

Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 272–275, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics MARS: A Specialized RTE System for Parser Evaluation Rui Wang† Yi Zhang†‡ † Department of Computational Linguistics, Saarland University ‡ LT-Lab, German Research Center for Artificial Intelligence Im Stadtwald, 66123 Saarbr¨ucken, Germany {rwang,yzhang}@coli.uni-sb.de Abstract inThis paper describes our participation the the SemEval-2010 Task #12, Parser Evaluation using Textual Entail- ment. Our system incorporated two depen- dency parsers, one semantic role labeler, and a deep parser based on hand-crafted grammars. The shortest path algorithm is applied on the graph representation of the parser outputs. Then, different types of features are extracted and the entail- ment recognition is casted into a machine- learning-based classification task. The best setting of the system achieves 66.78% of accuracy, which ranks the 3rd place. 1 Introduction The SemEval-2010 Task #12, Parser Evaluation using Textual Entailment (PETE) (Yuret et al., 2010), is an interesting task connecting two areas of research, parsing and recognizing textual entail- ment (RTE) (Dagan et al., 2005). The former is usually concerned with syntactic analysis in spe- cific linguistic frameworks, while the latter is be- lieved to involve more semantic aspects of the hu- man languages. However, no clear-cut boundary can be drawn between syntax and semantics for both tasks. In recent years, the parsing commu- nity has been reaching beyond what was usually accepted as syntactic structures. Many deep lin- guistic frameworks allow the construction of se- mantic representations in parallel to the syntactic structure. Meanwhile, data-driven shallow seman- tic parsers (or semantic role labelers) are another popular type of extension to enrich the information in the parser outputs. Although entailment is described as a semantic relation, RTE, in practice, covers linguistic phe- nomena at various levels, from surface text to the meaning, even to the context and discourse. One proposal of solving the problem is to deal with dif- ferent cases of entailment using different special- ized RTE modules (Wang and Neumann, 2009). Then, the PETE data can be naturally classified into the syntactic and shallow semantic categories. By participating in this shared task, we aim to investigate whether different parsing outputs leads to different RTE accuracy, and on the contrary, whether the “application”-based evaluation pro- vides insights to the parser comparison. Further, we investigate if strict grammaticality checking with a linguistic grammar is helpful in this task. 2 System Description The workflow of the system is shown in Figure 1 and the details of the three components will be elaborated on in the following sections. 2.1 Preprocessing In this paper, we generally refer all the linguistic analyses on the text as preprocessing. The output of this procedure is a graph representation, which approximates the meaning of the input text. In par- ticular, after tokenization and POS tagging, we did dependency parsing and semantic role labeling. In addition, HPSG parsing is a filter for ungrammat- ical hypotheses. Tokenization and POS Tagging We use the Penn Treebank style tokenization …

Content truncated to 500 words