Cross-document Event Coreference Resolution based on Cross-media Features

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 201–206, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Cross-document Event Coreference Resolution based on Cross-media Features Tongtao Zhang1, Hongzhi Li2, Heng Ji1, Shih-Fu Chang2 1Computer Science Department, Rensselaer Polytechnic Institute {zhangt13, jih}@rpi.edu 2Department of Computer Science, Columbia University {hongzhi.li, shih.fu.chang}@columbia.edu Abstract In this paper we focus on a new problem of event coreference resolution across televi- sion news videos. Based on the observa- tion that the contents from multiple data modalities are complementary, we develop a novel approach to jointly encode effec- tive features from both closed captions and video key frames. Experiment re- sults demonstrate that visual features pro- vided 7.2% absolute F-score gain on state- of-the-art text based event extraction and coreference resolution. 1 Introduction TV news is the medium that broadcasts events, stories and other information via television. The broadcast is conducted in programs with the name of “Newscast”. Typically, newscasts require one or several anchors who are introducing stories and coordinating transition among topics, reporters or journalists who are presenting events in the fields and scenes that are captured by cameramen. Sim- ilar to newspapers, the same stories are often re- ported by multiple newscast agents. Moreover, in order to increase the impact on audience, the same stories and events are reported for mutliple times. TV audience passively receives redundant infor- mation, and often has difficulty in obtaining clear and useful digest of ongoing events. These proper- ties lead to needs for automatic methods to cluster information and remove redundancy. We propose a new research problem of event coreference reso- lution across multiple news videos. To tackle this problem, a good starting point is processing the Closed Captions (CC) which is accompanying videos in newcasts. The CC is either generated by automatic speech recogni- tion (ASR) systems or transcribed by a human stenotype operator who inputs phonetics which are Figure 1: Similar visual contents improve detec- tion of a coreferential event pair which has a low text-based confidence score. Closed Captions: “It ’s not clear when it was killed.”; “Jordan just executed two ISIS prisoners, direct retaliation for the capture of the killing Jordanian pilot.” instantly and automatically translated into texts, where events can be extracted. There exist some previous event coreference resolution work such as (Chen and Ji, 2009b; Chen et al., 2009; Lee et al., 2012; Bejan and Harabagiu, 2010). However, they only focused on formally written newswire articles and utilized textual features. Such ap- proaches do not perform well on CC due to (1). the propagated errors from upper stream compo- nents (e.g., automatic speech/stenotype recogni- tion and event extraction); (2). the incomplete- ness of information. Different from written news, newscasts are often limited in time due to fixed TV program schedules, thus, anchors and journal- ists are trained and expected to organize reports 201 which are comprehensively informative with com- plementary visual and CC descriptions within a short time. These two sides have minimal over- lapped …

Content truncated to 500 words