Automatic Feature Engineering for Answer Selection and Extraction

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 458–467, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Automatic Feature Engineering for Answer Selection and Extraction Aliaksei Severyn DISI, University of Trento 38123 Povo (TN), Italy severyn@disi.unitn.it Alessandro Moschitti Qatar Computing Research Institue 5825 Doha, Qatar amoschitti@qf.org.qa Abstract This paper proposes a framework for automat- ically engineering features for two important tasks of question answering: answer sentence selection and answer extraction. We represent question and answer sentence pairs with lin- guistic structures enriched by semantic infor- mation, where the latter is produced by auto- matic classifiers, e.g., question classifier and Named Entity Recognizer. Tree kernels ap- plied to such structures enable a simple way to generate highly discriminative structural fea- tures that combine syntactic and semantic in- formation encoded in the input trees. We con- duct experiments on a public benchmark from TREC to compare with previous systems for answer sentence selection and answer extrac- tion. The results show that our models greatly improve on the state of the art, e.g., up to 22% on F1 (relative improvement) for answer ex- traction, while using no additional resources and no manual feature engineering. 1 Introduction Question Answering (QA) systems are typically built from three main macro-modules: (i) search and retrieval of candidate passages; (ii) reranking or se- lection of the most promising passages; and (iii) an- swer extraction. The last two steps are the most in- teresting from a Natural Language Processing view- point since deep linguistic analysis can be carried out as the input is just a limited set of candidates. Answer sentence selection refers to the task of se- lecting the sentence containing the correct answer among the different sentence candidates retrieved by a search engine. Answer extraction is a final step, required for factoid questions, consisting in extracting multi- words constituting the synthetic answer, e.g., Barack Obama for a question: Who is the US president? The definition of rules for both tasks is conceptually demanding and involves the use of syntactic and se- mantic properties of the questions and its related an- swer passages. For example, given a question from TREC QA1: Q: What was Johnny Appleseed’s real name? and a relevant passage, e.g., retrieved by a search engine: A: Appleseed, whose real name was John Chapman, planted many trees in the early 1800s. a rule detecting the semantic links between Johnny Appleseed’s real name and the correct answer John Chapman in the answer sentence has to be engineered. This requires the definition of other rules that associate the question pattern real name ?(X) with real name is(X) of the answer sentence. Although this can be done by an expert NLP engineer, the effort for achieving the necessary coverage and a reasonable accuracy is not negligible. An alternative to manual rule definition is the use of machine learning, which often shifts the problem 1We use it as our running example in the rest of the paper. 458 to the easier task …

Content truncated to 500 words