Time-Efficient Creation of an Accurate Sentence Fusion Corpus

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 317–320, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Time-Efficient Creation of an Accurate Sentence Fusion Corpus Kathleen McKeown, Sara Rosenthal, Kapil Thadani and Coleman Moore Columbia University New York, NY 10027, USA {kathy,sara,kapil}@cs.columbia.edu, cjm2140@columbia.edu Abstract Sentence fusion enables summarization and question-answering systems to produce out- put by combining fully formed phrases from different sentences. Yet there is little data that can be used to develop and evaluate fu- sion techniques. In this paper, we present a methodology for collecting fusions of simi- lar sentence pairs using Amazon’s Mechani- cal Turk, selecting the input pairs in a semi- automated fashion. We evaluate the results using a novel technique for automatically se- lecting a representative sentence from multi- ple responses. Our approach allows for rapid construction of a high accuracy fusion corpus. 1 Introduction Summarization and question-answering systems must transform input text to produce useful output text, condensing an input document or document set in the case of summarization and selecting text that meets the question constraints in the case of question answering. While many systems use sentence ex- traction to facilitate the task, this approach risks in- cluding additional, irrelevant or non-salient informa- tion in the output, and the original sentence wording may be inappropriate for the new context in which it appears. Instead, recent research has investigated methods for generating new sentences using a tech- nique called sentence fusion (Barzilay and McKe- own, 2005; Marsi and Krahmer, 2005; Filippova and Strube, 2008) where output sentences are generated by fusing together portions of related sentences. While algorithms for automated fusion have been developed, there is no corpus of human-generated fused sentences available to train and evaluate such systems. The creation of such a dataset could pro- vide insight into the kinds of fusions that people produce. Furthermore, since research in the related task of sentence compression has benefited from the availability of training data (Jing, 2000; Knight and Marcu, 2002; McDonald, 2006; Cohn and La- pata, 2008), we expect that the creation of this cor- pus might encourage the development of supervised learning techniques for automated sentence fusion. In this work, we present a methodology for cre- ating such a corpus using Amazon’s Mechanical Turk1, a widely used online marketplace for crowd- sourced task completion. Our goal is the generation of accurate fusions between pairs of sentences that have some information in common. To ensure that the task is performed consistently, we abide by the distinction proposed by Marsi and Krahmer (2005) between intersection fusion and union fusion. In- tersection fusion results in a sentence that contains only the information that the sentences had in com- mon and is usually shorter than either of the original sentences. Union fusion, on the other hand, results in a sentence that contains all information content from the original two sentences. An example of in- tersection and union fusion is shown in Figure 1. We solicit …

Content truncated to 500 words