CoMeT: Integrating different levels of linguistic modeling for meaning assessment

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 608–616, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics CoMeT: Integrating different levels of linguistic modeling for meaning assessment Niels Ott Ramon Ziai Michael Hahn Sonderforschungsbereich 833Detmar Meurers Eberhard Karls Universit¨at T¨ubingen {nott,rziai,mhahn,dm}@sfs.uni-tuebingen.de Abstract This paper describes the CoMeT system, our contribution to the SemEval 2013 Task 7 chal- lenge, focusing on the task of automatically assessing student answers to factual questions. CoMeT is based on a meta-classifier that uses the outputs of the sub-systems we developed: CoMiC, CoSeC, and three shallower bag ap- proaches. We sketch the functionality of all sub-systems and evaluate their performance against the official test set of the challenge. CoMeT obtained the best result (73.1% accu- racy) for the 3-way unseen answers in Beetle among all challenge participants. We also dis- cuss possible improvements and directions for future research. 1 Introduction Our contribution to the SemEval 2013 Task 7 chal- lenge (Dzikovska et al., 2013) presented here is based on our research in the A4 project1 of the SFB 833, which is dedicated to the question how meaning can be computationally compared in realistic situations. In realistic situations, utterances are not necessarily well-formed or complete, there may be individual differences in situative and world knowledge among the speakers. This can complicate or even preclude a complete linguistic analysis, leading us to the fol- lowing research question: Which linguistic repre- sentations can be used effectively and robustly for comparing the meaning of sentences and text frag- ments computationally? 1http://purl.org/dm/projects/sfb833-a4 In order to work on effective and robust processing, we base our work on reading comprehension exer- cises for foreign language learners, of which we are also collecting a large corpus (Ott et al., 2012). Our first system, CoMiC, is an alignment-based approach which exists in English and German variants (Meur- ers et al., 2011a; Meurers et al., 2011b). CoMiC uses various levels of linguistic abstraction from sur- face tokens to dependency parses. Further work that we are starting to tackle includes the utilization of Information Structure (Krifka, 2007) in the system. The second approach emerging from the research project is CoSeC (Hahn and Meurers, 2011; Hahn and Meurers, 2012), a semantics-based system for meaning comparison that was developed for German from the start and was ported to operate on English for this shared task. As a novel contribution in this paper, we present CoMeT (Comparing Meaning in T¨ubingen), a system that employs a meta-classifier for combining the output of CoMiC and CoSeC and three shallower bag approaches. In terms of the general context of our work, short answer assessment essentially comes in the two fla- vors of meaning comparison and grading, the first trying to determine whether or not two utterances convey the same meaning, the latter aimed at grading the abilities of students (cf. Ziai et al., 2012). Short answer assessment is also closely related to the field of Recognizing Textual Entailment (RTE, …

Content truncated to 500 words