Multi-document Summarisation and the PASCAL Textual Entailment Challenge

Proceedings of the Australasian Language Technology Workshop 2005, pages 215–223, Sydney, Australia, December 2005. Multi-document Summarisation and the PASCAL Textual Entailment Challenge Nicola Stokes NICTA Victoria Laboratory, Department of Computer Science and Software Engineering, University of Melbourne. nicola.stokes@nicta.com.au Eamonn Newman School of Computer Science and Informatics, University College Dublin, Ireland. eamonn.newman@ucd.ie Abstract A fundamental problem for systems that re- quire natural language understanding capabili- ties is the identification of instances of semantic equivalence and paraphrase in text. The PAS- CAL Recognising Textual Entailment (RTE) challenge is a recently proposed research ini- tiative that addressed this problem by provid- ing an evaluation framework for the develop- ment of generic “semantic engines” that can be used to identify language variability in a variety of applications such as Information Retrieval, Machine Translation and Question Answering. This paper discusses the suitability of the RTE evaluation datasets as a framework for evaluat- ing the problem of redundancy recognition in multi-document summarisation, i.e. the iden- tification of repetitive information across docu- ments. This paper also reports on the develop- ment of an additional dataset containing exam- ples of informationally equivalent sentence pairs that are typically found in machine generated summaries. The performance of a competitive entailment recognition system on this dataset is also reported. 1 Introduction The aim of multi–document summarisation (MDS) is to generate a concise and coherent summary given a cluster of related documents. Although this process is a natural extension of single–document summarisation, MDS poses a number of unique challenges such as, how to manage contradictory and repetitive informa- tion in the cluster, and how to order extracted information in the resultant summary. A pop- ular approach to the MDS problem is to first identify and cluster repetitive information units across documents, then select representative sentences from the “dominant” clusters, and finally generate an extractive summary from these sentences. This approach assumes, like many others in text summarisation, that the repetition of information is an indication of in- formation importance and consequently sum- mary relevancy. The simplest method for de- termining commonality across documents is to group text units (e.g. sentences, paragraphs) that exhibit a high concentration of word over- lap. However, this approximate method for recognising similar semantic content is often in- sufficient due to instances of language variabil- ity such as paraphrase and synonymy. Figure 1 shows two sentences (A and B) that are seman- tically equivalent but syntactically different. Text A: Agassi’s dream run is ended by world’s number one player. Text B: Federer beats Agassi. Figure 1: Paraphrases with minimal word over- lap In this paper we discuss the suitability of the recently proposed PASCAL Recognising Tex- tual Entailment (RTE) challenge (Dagan et al., 2005a) as an evaluation methodology for deter- mining the performance of redundant informa- tion identification techniques in the context of MDS. The aim of the RTE challenge is to aid the development of generic “semantic engines” that can be used in a number of applications such as Information Retrieval, Information Extraction, Text Summarisation and Machine …

Content truncated to 500 words