The IARPA BETTER Program Abstract Task Four New Semantically Annotated Corpora from IARPA’s BETTER Program

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3595–3600 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3595 The IARPA BETTER Program Abstract Task Four New Semantically Annotated Corpora from IARPA’s BETTER Program Carl Rubino, Timothy McKinnon Intelligence Advanced Research Projects Activity (IARPA) Bethesda, Maryland USA {Carl.Rubino, Timothy.McKinnon}@iarpa.gov Abstract IARPA’s Better Extraction from Text Towards Enhanced Retrieval (BETTER) Program created multiple multilingual datasets to spawn and evaluate cross-language information extraction and information retrieval research and development in zero-shot conditions. The first set of these resources for information extraction, the “Abstract” data will be released to the public at LREC 2022 in four languages to champion further information extraction work in this area. This paper presents the event and argument annotation in the Abstract Evaluation phase of BETTER, as well as the data collection, preparation, partitioning and mark-up of the datasets. Keywords: CLIR, CLIE, zero shot, human-in-the-loop 1. IARPA’s BETTER Program The Intelligence Advanced Research Projects Activity (IARPA) kicked off the BETTER program in Boston on October 2019 to advance research in multilingual, cross- lingual and zero-shot information extraction (IE) and information retrieval (IR). BETTER was designed as a 42- month program, broken into three phases of decreasing length (18/12/12 months), each with its own evaluation language(s) and topic domain(s). The program goal was to develop enhanced methods for personalized, multilingual semantic extraction and retrieval from multilingual newswire text. Four large teams (or “Performers”) were competitively selected to participate in the program based on their responses to a Broad Agency Announcement (BAA). The prime contractors for these teams were the University of Southern California Information Sciences Institute (USC-ISI), Brown University, Johns Hopkins University, and Raytheon BBN. Performers were contracted to develop systems that quickly and accurately extract complex semantic information from raw text documents in multiple languages in a way that is adaptive to the information needs of a specific monolingual English user. Systems were expected to leverage this extracted information to enable automatic IR and efficient human triage of relevant documents from massive stores of text documents. A major goal of the program was to incentivize the rapid adaptation of the technologies to new languages and domains with minimal effort. For each of the three phases of the program IE and IR from a specific surprise domain (e.g., corporate mergers and acquisitions, protests, government corruption, epidemiology, etc.) was levied on the Performers by IARPA’s Test and Evaluation Team. Within each phase, Performer systems were evaluated on three IE tasks and one human-in-the-loop (HITL) IR task. Within each phase, the initial task focused on extraction of Abstract events, which we describe in Section 2, while the second and third tasks focused on extraction of events comprising increasingly granular semantic information (so-called Basic and Granular events), previously referred to as “fine- grained” and “finer grained” events in Beieler (2018). A major technical challenge posed by BETTER is that Performer systems must carry out ‘zero-shot’ learning: Performers are only provided training data in …

Content truncated to 500 words