Multilingual and cross-lingual news topic tracking

Multilingual and cross-lingual news topic tracking Bruno Pouliquen, Ralf Steinberger, Camelia Ignat, Emilia Käsper & Irina Temnikova Joint Research Centre, European Commission T.P. 267, Via E. Fermi 1 21020 Ispra (VA), Italy http://www.jrc.it/langtech Firstname.Lastname@jrc.it Abstract We are presenting a working system for automated news analysis that ingests an average total of 7600 news articles per day in five languages. For each language, the system detects the major news stories of the day using a group-average unsupervised ag- glomerative clustering process. It also tracks, for each cluster, related groups of articles published over the previous seven days, using a cosine of weighted terms. The system furthermore tracks re- lated news across languages, in all language pairs involved. The cross-lingual news cluster similarity is based on a linear combination of three types of input: (a) cognates, (b) automatically detected ref- erences to geographical place names and (c) the re- sults of a mapping process onto a multilingual clas- sification system. A manual evaluation showed that the system produces good results. 1 Introduction Most large organisations, companies and politi- cal parties have a department analysing the news on a daily basis. Motivations differ, but often these organisations want to know how they and their leading members are represented in the news, or they need to know whether there has been any event they ought to know about. Examples of ex- isting news gathering and analysis systems are In- formedia1 and the Europe Media Monitor (Best et al. 2002). DARPA has taken an interest in the do- main and launched, in 1996, the Topic Detection and Tracking task2 (TDT) under the TIDES pro- gram. It distinguishes three major tasks: (a) seg- mentation of a continuous information flow (e.g. spoken news) into individual news items, (b) de- tection of breaking news, i.e. of a new subject that has not previously been discussed, and (c) topic tracking, i.e. the identification of related news over time. Our task is the analysis of a multilingual col- lection of written news articles, which means that segmentation (task a) is of no relevance. Neither do we present here work on the detection of new 1 http://www.informedia.cs.cmu.edu/ 2 http://www.nist.gov/speech/tests/tdt/ topics (task b). Instead, we focus on the topic tracking task (c), and especially on the novel as- pect of cross-lingual tracking. The aim of our work is to provide an automati- cally generated overview over the major news of each day (midnight to midnight) in the languages English, German, French, Spanish and Italian. The corpus consists of news items gathered from a large number of internet news sites world-wide, and of various subscription news wires (Best et al. 2002). The texts are thus from hundreds of differ- ent sources (feeds) which often discuss the same events. Newspapers often publish the news they receive from press agencies with no or few amendments. The corpus of news articles thus con- tains not only summaries of the same events writ- ten by different journalists, but also many dupli- cates and near duplicates of the …

Content truncated to 500 words