Presenting a New Dataset for the Timeline Generation Problem

Presenting a New Dataset for the Timeline Generation Problem Xavier Holt University of Sydney xhol4115@ uni.sydney.edu.au Will Radford Hugo.ai wradford@ hugo.ai Ben Hachey University of Sydney ben.hachey@ sydney.edu.au Abstract The timeline generation task summarises an entity’s biography by selecting stories representing key events from a large pool of relevant documents. This paper ad- dresses the lack of a standard dataset and evaluative methodology for the problem. We present and make publicly available a new dataset of 18,793 news articles cover- ing 39 entities. For each entity, we provide a gold standard timeline and a set of entity- related articles. We propose ROUGE as an evaluation metric and validate our dataset by showing that top Google results outper- form straw-man baselines. 1 Introduction Information is more readily available in greater quantities than ever before. Timeline generation is a recent method for summarising data – tak- ing a large pool of entity-related documents as in- put and selecting a small set that best describe the key events in the entity’s life. There are several challenges in evaluation: (1) finding gold-standard timelines, (2) finding corpora from which to draw documents to build timelines, and (3) evaluating system timelines. Standard practice for the first challenge is to make use of existing timelines produced by news agencies (Chieu and Lee, 2004; Yan et al., 2011a), but these are constrained by the tight editorial fo- cus on prominent entities and depends on well- funded news agencies. Another approach is to an- notate new timelines from the web for domains of choice. Wang (2013) do this, but do not make their data available for direct comparison. Regard- ing the second challenge, access to the document pool used during the annotation process is also im- portant, as any system must have a reasonable set from which to choose. Previous approaches have used drawn on work- ing in document summarisation, using ROUGE (Lin, 2004) to evaluate timeline generation (Chieu and Lee, 2004; Yan et al., 2011a; Yan et al., 2011b; Ahmed and Xing, 2012; Wang, 2013). This is con- venient as each element in a timeline can repre- sent a story which can be equivalently described by many different documents. However, previous work has not validated the use of ROUGE for eval- uating timeline generation. We present a general framework for creating a crowd-sourced datasets for evaluating timeline generation, including choosing a set of entities, deriving articles for annotation from Wikipedia, and annotating these articles to generate a gold standard. The dataset covers a broad range of entities with different levels of news-coverage or public- ity. We provide gold-standard timelines for each entity, as well as a larger pool of topically-linked documents for further development. We anal- yse the dataset, showing some interesting arti- facts of crowd-worker importance judgements and use ROUGE evaluation to verify that the crowd- annotations correlate well with Google News1 rankings. This reflects favourably on Google News, suggesting that it is a strong baseline for timeline generation. We release the dataset gener- ated through this …

Content truncated to 500 words