An Empirical Study of Information Synthesis Task

An Empirical Study of Information Synthesis Tasks Enrique Amig´o Julio Gonzalo V´ıctor Peinado Anselmo Pe˜nas Felisa Verdejo Departamento de Lenguajes y Sistemas Inform´aticos Universidad Nacional de Educaci´on a Distancia c/Juan del Rosal, 16 - 28040 Madrid - Spain {enrique,julio,victor,anselmo,felisa}@lsi.uned.es Abstract This paper describes an empirical study of the “In- formation Synthesis” task, defined as the process of (given a complex information need) extracting, or- ganizing and inter-relating the pieces of information contained in a set of relevant documents, in order to obtain a comprehensive, non redundant report that satisfies the information need. Two main results are presented: a) the creation of an Information Synthesis testbed with 72 reports manually generated by nine subjects for eight com- plex topics with 100 relevant documents each; and b) an empirical comparison of similarity metrics be- tween reports, under the hypothesis that the best metric is the one that best distinguishes between manual and automatically generated reports. A met- ric based on key concepts overlap gives better re- sults than metrics based on n-gram overlap (such as ROUGE) or sentence overlap. 1 Introduction A classical Information Retrieval (IR) system helps the user finding relevant documents in a given text collection. In most occasions, however, this is only the first step towards fulfilling an information need. The next steps consist of extracting, organizing and relating the relevant pieces of information, in or- der to obtain a comprehensive, non redundant report that satisfies the information need. In this paper, we will refer to this process as In- formation Synthesis. It is normally understood as an (intellectually challenging) human task, and per- haps the Google Answer Service1 is the best gen- eral purpose illustration of how it works. In this ser- vice, users send complex queries which cannot be answered simply by inspecting the first two or three documents returned by a search engine. These are a couple of real, representative examples: a) I’m looking for information concerning the history of text compression both before and with computers. 1http://answers.google.com b) Provide an analysis on the future of web browsers, if any. Answers to such complex information needs are provided by experts which, commonly, search the Internet, select the best sources, and assemble the most relevant pieces of information into a report, organizing the most important facts and providing additional web hyperlinks for further reading. This Information Synthesis task is understood, in Google Answers, as a human task for which a search engine only provides the initial starting point. Our mid- term goal is to develop computer assistants that help users to accomplish Information Synthesis tasks. From a Computational Linguistics point of view, Information Synthesis can be seen as a kind of topic-oriented, informative multi-document sum- marization, where the goal is to produce a single text as a compressed version of a set of documents with a minimum loss of relevant information. Un- like indicative summaries (which help to determine whether a document is relevant to a particular topic), informative summaries must be helpful to answer, for instance, …

Content truncated to 500 words