A Study in Contradiction: Data and Annotation for AIDA Focusing on Informational Conflict in Russia-Ukraine Relations

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1831–1838 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1831 A Study in Contradiction: Data and Annotation for AIDA Focusing on Informational Conflict in Russia-Ukraine Relations Jennifer Tracey, Ann Bies, Jeremy Getman, Kira Griffitt, Stephanie Strassel Linguistic Data Consortium, University of Pennsylvania 3600 Market Street, Suite 810, Philadelphia, PA 19104 USA {garjen, bies, jgetman, kiragrif, strassel}@ldc.upenn.edu Abstract This paper describes data resources created for Phase 1 of the DARPA Active Interpretation of Disparate Alternatives (AIDA) program, which aims to develop language technology that can help humans manage large volumes of sometimes conflicting information to develop a comprehensive understanding of events around the world, even when such events are described in multiple media and languages. Especially important is the need for the technology to be capable of building multiple hypotheses to account for alternative interpretations of data imbued with informational conflict. The corpus described here is designed to support these goals. It focuses on the domain of Russia-Ukraine relations and contains multimedia source data in English, Russian and Ukrainian, annotated to support development and evaluation of systems that perform extraction of entities, events, and relations from individual multimedia documents, aggregate the information across documents and languages, and produce multiple “hypotheses” about what has happened. This paper describes source data collection, annotation, and assessment. Keywords: information extraction, disinformation, multimedia corpora 1. Introduction The DARPA Active Interpretation of Disparate Alternatives (AIDA) program aims toward the development of technology that can assist in cultivating and maintaining understanding of events in a world where the truth can be hard to establish given conflicting accounts of what happened (e.g. who did what to whom and/or where and when events occurred). This is made more challenging when information relevant to obtaining a comprehensive understanding of what happened is scattered across different languages and media types. To address this challenge, AIDA systems must extract entities, events, and relations from individual multimedia documents, aggregate that information across documents and languages into a coherent semantic representation, and produce multiple knowledge graph hypotheses that characterize the conflicting accounts about what actually happened that are present in the corpus (DARPA, 2107) . To support AIDA system development and evaluation, LDC produces corpora of source data and annotations and assesses system responses. Each phase of the program concentrates on a different topic scenario, with a source corpus, annotation tasks and assessment approach reflecting the particular technology goals for that phase. The current paper describes the resources produced for Phase 1 (as well as pilot work leading up to Phase 1), whose scenario is political relations between Russia and Ukraine in the 2010s. Multimedia source data was collected in English, Russian, and Ukrainian, focusing on several subtopics within the Russia-Ukraine domain that were known to have informational conflict; annotations were performed on a subset of the documents, including annotation of entities, relations and events, cross document coreference of annotated entities and events, and annotation of which …

Content truncated to 500 words