Answer Extraction as Sequence Tagging with Tree Edit Distance

Proceedings of NAACL-HLT 2013, pages 858–867, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Answer Extraction as Sequence Tagging with Tree Edit Distance Xuchen Yao and Benjamin Van Durme Johns Hopkins University Baltimore, MD, USA Chris Callison-Burch∗ University of Pennsylvania Philadelphia, PA, USA Peter Clark Vulcan Inc. Seattle, WA, USA Abstract Our goal is to extract answers from pre- retrieved sentences for Question Answering (QA). We construct a linear-chain Conditional Random Field based on pairs of questions and their possible answer sentences, learning the association between questions and answer types. This casts answer extraction as an an- swer sequence tagging problem for the first time, where knowledge of shared structure be- tween question and source sentence is incor- porated through features based on Tree Edit Distance (TED). Our model is free of man- ually created question and answer templates, fast to run (processing 200 QA pairs per sec- ond excluding parsing time), and yields an F1 of 63.3% on a new public dataset based on prior TREC QA evaluations. The developed system is open-source, and includes an imple- mentation of the TED model that is state of the art in the task of ranking QA pairs. 1 Introduction The success of IBM’s Watson system for Question Answering (QA) (Ferrucci et al., 2010) has illus- trated a continued public interest in this topic. Wat- son is a sophisticated piece of software engineering consisting of many components tied together in a large parallel architecture. It took many researchers working full time for years to construct. Such re- sources are not available to individual academic re- searchers. If they are interested in evaluating new ideas on some aspect of QA, they must either con- struct a full system, or create a focused subtask ∗Performed while faculty at Johns Hopkins University. paired with a representative dataset. We follow the latter approach and focus on the task of answer ex- traction, i.e., producing the exact answer strings for a question. We propose the use of a linear-chain Conditional Random Field (CRF) (Lafferty et al., 2001) in or- der to cast the problem as one of sequence tagging by labeling each token in a candidate sentence as ei- ther Beginning, Inside or Outside (BIO) of an an- swer. This is to our knowledge the first time a CRF has been used to extract answers.1 We uti- lize not only traditional contextual features based on POS tagging, dependency parsing and Named Entity Recognition (NER), but most importantly, features extracted from a Tree Edit Distance (TED) model for aligning an answer sentence tree with the ques- tion tree. The linear-chain CRF, when trained to learn the associations between question and answer types, is a robust approach against error propaga- tion introduced in the NLP pipeline. For instance, given an NER tool that always (i.e., in both train- ing and test data) recognizes the pesticide DDT as an ORG, our model realizes, when a question is asked about the type of chemicals, the correct an- swer might …

Content truncated to 500 words