The Answer is at your Fingertips: Improving Passage Retrieval for Web Question Answering with Search Behavior Data

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1011–1021, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics The Answer is at your Fingertips: Improving Passage Retrieval for Web Question Answering with Search Behavior Data Mikhail Ageev∗ Moscow State University mageev@yandex.ru Dmitry Lagun Emory University dlagun@emory.edu Eugene Agichtein Emory University eugene@mathcs.emory.edu Abstract Passage retrieval is a crucial first step of au- tomatic Question Answering (QA). While ex- isting passage retrieval algorithms are effec- tive at selecting document passages most sim- ilar to the question, or those that contain the expected answer types, they do not take into account which parts of the document the searchers actually found useful. We propose, to the best of our knowledge, the first success- ful attempt to incorporate searcher examina- tion data into passage retrieval for question an- swering. Specifically, we exploit detailed ex- amination data, such as mouse cursor move- ments and scrolling, to infer the parts of the document the searcher found interesting, and then incorporate this signal into passage re- trieval for QA. Our extensive experiments and analysis demonstrate that our method signif- icantly improves passage retrieval, compared to using textual features alone. As an addi- tional contribution, we make available to the research community the code and the search behavior data used in this study, with the hope of encouraging further research in this area. 1 Introduction Automated Question Answering (QA), is an attrac- tive variation of search where the QA system auto- matically returns an answer to a user’s question, in- stead of a list of document results. Passage retrieval is a first critical step of QA system, where candi- date passages are identified and scored as likely to contain an answer. While significant progress has been made recently on incorporating syntactic and semantic analysis for improving the QA system per- formance, this analysis is typically applied only on the (limited) set of candidate passages retrieved. The main reason is that it is generally not practical to perform deep analysis on all documents in a large collection, and not yet feasible for the Web at large. ∗Work done at Emory University. In the web search setting, automated question answering presents additional challenges and op- portunities. On the downside, the questions and queries from real users are often not grammatical or well-formed, differing from the questions used in the traditional TREC Question Answering evalua- tions (Kelly and Lin, 2007; Sun et al., 2005). On the upside, by interacting with a search engine, the mil- lions of searchers implicitly provide additional clues about usefulness of documents, result ranking, and other aspects of the search process. In this paper, we explore making use of the search behavior data to improve passage retrieval for automated Question Answering on the web. Our basic observation is that when a user is at- tempting to answer a question, he or she will more carefully examine the parts of the document that contain an answer. This observation is intuitive, and is …

Content truncated to 500 words