Finding What Matters in Questions

Proceedings of NAACL-HLT 2013, pages 878–887, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Finding What Matters in Questions Xiaoqiang Luo, Hema Raghavan, Vittorio Castelli, Sameer Maskey and Radu Florian IBM T.J. Watson Research Center 1101 Kitchawan Road, Yorktown Heights, NY 10598 {xiaoluo,hraghav,vittorio,smaskey,raduf}@us.ibm.com Abstract In natural language question answering (QA) systems, questions often contain terms and phrases that are critically important for re- trieving or finding answers from documents. We present a learnable system that can ex- tract and rank these terms and phrases (dubbed mandatory matching phrases or MMPs), and demonstrate their utility in a QA system on In- ternet discussion forum data sets. The system relies on deep syntactic and semantic analysis of questions only and is independent of rele- vant documents. Our proposed model can pre- dict MMPs with high accuracy. When used in a QA system features derived from the MMP model improve performance significantly over a state-of-the-art baseline. The final QA sys- tem was the best performing system in the DARPA BOLT-IR evaluation. 1 Introduction In most question answering (QA) systems and search engines term-weights are assigned in a con- text independent fashion using simple TF-IDF like models (Robertson and Walker, 1994; Ponte and Croft, 1998). Even the more recent advances in information retrieval techniques for query term weighting (Bendersky et al., 2010; Bendersky, 2011) typically rely on bag-of-words models and cor- pus statistics, such as inverse-document-frequency (IDF), to assign weights to terms in questions. While such solutions may work for keyword queries of the type common on search engines such as Google, they do not exploit syntactic and semantic informa- tion when it comes to well formed natural language questions. In this paper we propose a new model that identifies important terms and phrases in a natu- ral language question, providing better query analy- sis that ultimately leads to significant improvements in a QA system. To motivate the work presented here, consider the query “How does one apply for a New York day care license?”. A bag-of-words model would likely as- sign a high score to “New licenses for day care cen- ters in York county, PA” because of high word over- lap, but it does not answer the question, and also the state is wrong. A matching component that uses the phrases “New York,” “day care,” and “license” is likely to do better. However, a better matching component will understand that in the context of this query all three phrases “New York,” “day care” and “license” are important, and that “New York” needs to modify “day care.” A snippet that does not con- tain1 these important phrases, is unlikely an answer. We call these important phrases mandatory match- ing phrases (MMPs). In this paper, we explore deep syntactic and se- mantic analyses of questions to determine and rank MMPs. Unlike existing work (Zhao and Callan, 2010; Bendersky et al., 2010; Bendersky, 2011), where term/concept weights are learned from a set of questions and judged documents based on corpus- based statistics, we …

Content truncated to 500 words