Dataset Reproducibility and IR Methods in Timeline Summarization

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1763–1771 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1763 Dataset Reproducibility and IR Methods in Timeline Summarization Leo Born, Maximilian Bacher, Katja Markert Department of Computational Linguistics Heidelberg University 69120 Heidelberg, Germany {born, bacher, markert}@cl.uni-heidelberg.de Abstract Timeline summarization (TLS) generates a dated overview of real-world events based on event-specific corpora. The two standard datasets for this task were collected using Google searches for news reports on given events. Not only is this IR method not reproducible at different search times, it also uses components (such as document popularity) that are not always available for any large news corpus. It is unclear how TLS algorithms fare when provided with event corpora collected with varying IR methods. We therefore construct event-specific corpora from a large static background corpus, the newsroom dataset, using differing, relatively simple IR methods based on raw text alone. We show that the choice of IR method plays a crucial role in the performance of various TLS algorithms. A weak TLS algorithm can even match a stronger one by employing a stronger IR method in the data collection phase. Furthermore, the results of TLS systems are often highly sensitive to additional sentence filtering. We consequently advocate for integrating IR into the development of TLS systems and having a common static background corpus for evaluation of TLS systems. Keywords: timeline summarization, dataset construction, reproducibility 1. Introduction When real-world events get reported on, coverage can be quite exhaustive. This can either be due to the importance of the event or the duration of it. Timelines (see Table 1) provide a brief, dated overview of such events. In order to efficiently construct a timeline for a given event, timeline summarization (TLS) systems can be employed. Timeline summarization is a task with similarities to multi- document summarization (MDS). However, solving TLS requires generating a temporal framework for the summary, i.e. important dates have to be identified and corresponding date summaries have to be generated. In addition, the num- ber of input documents for TLS is normally an order of magnitude larger than for traditional MDS systems (hun- dreds instead of tens). TLS systems use two input components for a given event E (such as the BP oil spill): an input corpus CE that contains news documents covering E and human-written timelines TE (such as the one in Table 1) for evaluation (and possi- bly training). To constrain CEs to news articles that only cover E, various IR methods have been employed. To this end, multiple corpora for evaluation of the task have been created over the past decade. Unfortunately, most of the re- sulting corpora are not publicly available. In addition, the only two publicly available corpora TL17 and crisis (Tran et al., 2013b; Tran et al., 2015a) have been collected via the use of a commercial search engine at a given point in time. This means that the method is not reproducible and …

Content truncated to 500 words