Finding Good Enough: A Task-Based Evaluation of Query Biased Summarization for Cross-Language Information Retrieval

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 657–669, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Finding Good Enough: A Task-Based Evaluation of Query Biased Summarization for Cross Language Information Retrieval Jennifer Williams, Sharon Tam, Wade Shen MIT Lincoln Laboratory Human Language Technology Group 244 Wood Street, Lexington, MA 02420 USA jennifer.williams@ll.mit.edu, sharontam@alum.mit.edu swade@ll.mit.edu Abstract In this paper we present our task-based evaluation of query biased summarization for cross-language information retrieval (CLIR) using relevance prediction. We de- scribe our 13 summarization methods each from one of four summarization strate- gies. We show how well our methods perform using Farsi text from the CLEF 2008 shared-task, which we translated to English automtatically. We report preci- sion/recall/F1, accuracy and time-on-task. We found that different summarization methods perform optimally for different evaluation metrics, but overall query bi- ased word clouds are the best summariza- tion strategy. In our analysis, we demon- strate that using the ROUGE metric on our sentence-based summaries cannot make the same kinds of distinctions as our evalu- ation framework does. Finally, we present our recommendations for creating much- needed evaluation standards and datasets. 1 Introduction Despite many recent advances in query biased summarization for cross-language information re- trieval (CLIR), there are no existing evaluation standards or datasets to make comparisons among different methods, and across different languages (Tombros and Sanderson, 1998; Pingali et al., 2007; McCallum et al., 2012; Bhaskar and Bandy- opadhyay, 2012). Consider that creating this kind of summary requires familiarity with tech- niques from machine translation (MT), summa- rization, and information retrieval (IR). In this This work was sponsored by the Federal Bureau of Inves- tigation under Air Force Contract FA8721-05-C-0002. Opin- ions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the United States Government. paper, we arrive at the intersection of each of these research areas. Query biased summariza- tion (also known as query-focused, query-relevant, and query-dependent) involves automatically cap- turing relevant ideas and content from a document with respect to a given query, and presenting it as a condensed version of the original document. This kind of summarization is mostly used in search en- gines because when search results are tailored to a user’s information need, the user can find texts that they are looking for more quickly and more ac- curately (Tombros and Sanderson, 1998; Mori et al., 2004). Query biased summarization is a valu- able research area in natural language processing (NLP), especially for CLIR. Users of CLIR sys- tems meet their information needs by submitting their queries in L1 to search through documents that have been composed in L2, even though they may not be familiar with L2 (Hovy et al., 1999; Pingali et al., 2007). There are no standards for objectively evaluat- ing summaries for CLIR – a research gap that we begin to address in this paper. The problem we explore is two-fold: what kinds of summaries are well-suited for CLIR applications, and …

Content truncated to 500 words