Cross-media Structured Common Space for Multimedia Event Extraction

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2557–2568 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2557 Cross-media Structured Common Space for Multimedia Event Extraction Manling Li1∗, Alireza Zareian2∗, Qi Zeng1, Spencer Whitehead1, Di Lu3, Heng Ji1, Shih-Fu Chang2 1University of Illinois at Urbana-Champaign, 2Columbia University 3Dataminr {manling2,hengji}@illinois.edu, {az2407,sc250}@columbia.edu Abstract We introduce a new task, MultiMedia Event Extraction (M2E2), which aims to extract events and their arguments from multime- dia documents. We develop the first bench- mark and collect a dataset of 245 multi- media news articles with extensively anno- tated events and arguments.1 We propose a novel method, Weakly Aligned Structured Embedding (WASE), that encodes structured representations of semantic information from textual and visual data into a common em- bedding space. The structures are aligned across modalities by employing a weakly su- pervised training strategy, which enables ex- ploiting available resources without explicit cross-media annotation. Compared to uni- modal state-of-the-art methods, our approach achieves 4.0% and 9.8% absolute F-score gains on text event argument role labeling and visual event extraction. Compared to state- of-the-art multimedia unstructured representa- tions, we achieve 8.3% and 5.0% absolute F- score gains on multimedia event extraction and argument role labeling, respectively. By utiliz- ing images, we extract 21.4% more event men- tions than traditional text-only methods. 1 Introduction Traditional event extraction methods target a sin- gle modality, such as text (Wadden et al., 2019), images (Yatskar et al., 2016) or videos (Ye et al., 2015; Caba Heilbron et al., 2015; Soomro et al., 2012). However, the practice of contemporary journalism (Stephens, 1998) distributes news via multimedia. By randomly sampling 100 multi- media news articles from the Voice of America (VOA), we find that 33% of images in the arti- cles contain visual objects that serve as event ar- guments and are not mentioned in the text. Take ∗These authors contributed equally to this work. 1Our data and code are available at http://blender. cs.illinois.edu/software/m2e2 Figure 1: An example of Multimedia Event Extraction. An event mention and some event arguments (Agent and Person) are extracted from text, while the vehicle arguments can only be extracted from the image. Figure 1 as an example, we can extract the Agent and Person arguments of the Movement.Transport event from text, but can extract the Vehicle argu- ment only from the image. Nevertheless, event extraction is independently studied in Computer Vision (CV) and Natural Language Processing (NLP), with major differences in task definition, data domain, methodology, and terminology. Mo- tivated by the complementary and holistic na- ture of multimedia data, we propose MultiMedia Event Extraction (M2E2), a new task that aims to jointly extract events and arguments from multiple modalities. We construct the first benchmark and evaluation dataset for this task, which consists of 245 fully annotated news articles. We propose the first method, Weakly Aligned Structured Embedding (WASE), for extracting events and arguments from multiple modalities. Complex event structures have not been cov- ered by existing multimedia representation meth- ods (Wu …

Content truncated to 500 words