Bigger versus Similar: Selecting a Background Corpus for First Story Detection Based on Distributional Similarity

Proceedings of Recent Advances in Natural Language Processing, pages 1312–1320, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_150 1312 Bigger versus Similar: Selecting a Background Corpus for First Story Detection Based on Distributional Similarity Fei Wang and Robert J. Ross and John D. Kelleher Technological University Dublin ADAPT Research Centre d13122837@mydit.ie {robert.ross, john.d.kelleher}@dit.ie Abstract The current state of the art for First Story Detection (FSD) are nearest neighbour- based models with traditional term vector representations; however, one challenge faced by FSD models is that the docu- ment representation is usually defined by the vocabulary and term frequency from a background corpus. Consequently, the ideal background corpus should arguably be both large-scale to ensure adequate term coverage, and similar to the target do- main in terms of the language distribution. However, given these two factors cannot always be mutually satisfied, in this pa- per we examine whether the distributional similarity of common terms is more im- portant than the scale of common terms for FSD. As a basis for our analysis we propose a set of metrics to quantitatively measure the scale of common terms and the distributional similarity between cor- pora. Using these metrics we rank differ- ent background corpora relative to a target corpus. We also apply models based on different background corpora to the FSD task. Our results show that term distri- butional similarity is more predictive of good FSD performance than the scale of common terms; and, thus we demonstrate that a smaller recent domain-related cor- pus will be more suitable than a very large- scale general corpus for FSD. 1 Introduction Given a stream of documents about news events in a chronological order, the goal of First Story De- tection (FSD) is to identify the very first story for each event. Each story is processed in sequence, and a decision is made for a given candidate doc- ument on whether or not it discusses an event that has not been seen in previous documents; crucially this decision is made after processing the candi- date document but before processing any subse- quent documents (Allan et al., 1998; Yang et al., 1998). The decision making process for each in- coming document is normally based on a novelty score; namely, if the novelty score of a new docu- ment is higher than a given threshold, we say it is a first story. Hundreds of FSD models have been proposed in prior research, and the nearest neighbour-based models, in which the novelty score is defined as the distance from the new story to the closest ex- isting story, remain the state of the art (Wang et al., 2018). In the implementation of nearest neighbour-based FSD models, the first step is to represent each story with a sound document repre- sentation. Even though many deep learning-based document representations have been shown to achieve very good results in a range of NLP (Natu- ral Language Processing) tasks (Goldberg, 2017), the dominant document representation model for FSD remains the traditional term vector models in which …

Content truncated to 500 words