Bridging the Gap between Intrinsic and Perceived Relevance in Snippet Generation

Proceedings of COLING 2012: Technical Papers, pages 1129–1146, COLING 2012, Mumbai, December 2012. Bridging the Gap between Intrinsic and Perceived Relevance in Snippet Generation Jing He PabloDuboue Jian-YunNie DIRO, University of Montreal hejing@iro.montreal.ca, dubouep@iro.umontreal.ca, nie@iro.umontreal.ca ABSTRACT Snippet generation plays an important role in a search engine. Good snippets provide users a good indication on the main content of a search result related to the query and on whether one can find relevant information in it. Previous studies on snippet generation focused on selecting sentences that are related to the query and to the document. However, resulting snippet may look highly relevant while the document itself is not. A missing factor that has not been considered is the consistency between the perceived relevance by the user in reading the snippet and the intrinsic relevance of the document. This factor is important to avoid generating a seemingly relevant snippet for an irrelevant document and vice versa. In this paper, we incorporate this factor in a snippet generation method that imposes the constraint that the snippet of a more relevant document should also be more relevant to the query. We derive a set of pairwise preferences between sentences from relevance judgments. We then use this set to train a gradient boosting decision tree to model a sentence scoring function used in snippet generation. Compared to the existing snippet generation methods and to the snippets generated by a commercial search engine, our snippets are more consistent with the true relevance of the documents. When the snippets are incorporated into a document ranking function, we also observe a significant improvement in retrieval effectiveness. This study shows the importance to generate snippets indicating the right level of relevance to the search results. KEYWORDS: Search Engine Snippet, Query-biased Summarization, Document Retrieval. 1129 1 Introduction The quality of a search engine is not only determined by its document ranking function, but also by the way the search results presented to the user (Turpin et al., 2009). In par- ticular, good snippets can help the user select relevant documents to click on. Snippets can be considered as a type of extractive summary of the Web pages. It is thus important that a snippet reflects the main part of the document content related to the query topic. This is the traditional role that one assigns to a snippet. To this end, two main categories of approaches have been proposed for snippet generation: sentence retrieval (Sanderson, 1998; Han et al., 2000; Ko et al., 2007; Park and An, 2010; Daumé and Marcu, 2006) and query-biased document summarization (Conroy et al., 2006; Tombros and Sanderson, 1998; Varadarajan and Hristidis, 2005; Otterbacher et al., 2005; Wang et al., 2007; Kanungo et al., 2009). The sentence retrieval approach finds the most relevant sentences (or text fragments) from a document and uses them as the snippet. One drawback of this method is that it does not con- sider the fidelity of the sentence to the content of the document, i.e., the extracted sentences may not …

Content truncated to 500 words