CoMiC: Adapting a Short Answer Assessment System for Answer Selection

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 247–251, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics CoMiC: Adapting a Short Answer Assessment System for Answer Selection Bj¨orn Rudzewitz Ramon Ziai Sonderforschungsbereich 833 Eberhard Karls Universit¨at T¨ubingen Nauklerstraße 35 72070 T¨ubingen, Germany {brzdwtz,rziai}@sfs.uni-tuebingen.de Abstract Open forum threads exhibit a great variabil- ity in the quality and quantity of the answers they attract, making it difficult to manually moderate and separate relevant from irrelevant content. The goal of SemEval 2015 Task 3 (Subtask A, English) is to build systems that automatically distinguish between relevant and irrelevant content in forum threads. We extend a short answer assessment system to build relations between forum questions and an- swers with respect to similarity, question type, and answer content. The features are used in a sequence classifier to account for the conver- sation character of threads. The performance of this approach is modest in comparison to the other task participants and also to the per- formance the system usually reaches in short answer assessment. However, the new features implemented for this task are a first step in developing more fine-grained question-answer features and identifying relevant answers. 1 Introduction In this paper, we discuss the adaptation of our Short Answer Assessment (SAA) system CoMiC (Meurers et al., 2011) to Task 3, Subtask A (English) of Sem- Eval 2015, Answer Selection in Community Question Answering. The aim in the task was to distinguish helpful from unhelpful answers in a community fo- rum given a question. We enter the QA landscape from the perspective of evaluating student answers to reading comprehen- sion questions with respect to whether they contain the targeted content. In such settings, one generally has a reference answer to which a candidate answer can be compared, making alignment-based systems a natural solution. This is not the case for QA, where a system has to select or rank candidate answers with regard to a question posed. However, the present task is still interesting to us because it shares a central characteristic with SAA: one needs to identify the relevant part of an answer, given a question. In theo- retical linguistics, that relevant part is usually called focus (cf., e.g., Krifka (2007)), and several research groups have made efforts to annotate it in corpus data (Hajiˇcov´a and Sgall, 2001; Ritz et al., 2008; Calhoun et al., 2010; Ziai and Meurers, 2014). Automatic approaches to identifying focus have however yet to be proposed, so for the current task, we adapted and used our SAA system to align can- didate answers with the forum question, identifying whether and how question material was picked up, which in turn should indicate whether answers are on-topic. We then used a number of features to char- acterize the unaligned answer material, from POS classes to temporal expressions. We also encoded which question words were present in the question in the hope that the resulting classifier would pick up connections between individual question words and the different answer …

Content truncated to 500 words