Towards Few-Shot Event Mention Retrieval: An Evaluation Framework and A Siamese Network Approach

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1747–1752 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1747 Towards Few-Shot Event Mention Retrieval: An Evaluation Framework and A Siamese Network Approach Bonan Min, Yee Seng Chan, Lingjun Zhao Raytheon BBN Technologies 10 Moulton Street, Cambridge, MA 02138 {bonan.min, yeeseng.chan, lingjun.zhao}@raytheon.com Abstract Automatically analyzing events in a large amount of text is crucial for situation awareness and decision making. Previous approaches treat event extraction as "one size fits all" with an ontology defined a priori. The resulted extraction models are built just for extracting those types in the ontology. These approaches cannot be easily adapted to new event types nor new domains of interest. To accommodate personalized event-centric information needs, this paper introduces the few-shot Event Mention Retrieval (EMR) task: given a user-supplied query consisting of a handful of event mentions, return relevant event mentions found in a corpus. This formulation enables "query by example", which drastically lowers the bar of specifying event-centric information needs. The retrieval setting also enables fuzzy search. We present an evaluation framework leveraging existing event datasets such as ACE. We also develop a Siamese Network approach, and show that it performs better than ad-hoc retrieval models in the few-shot EMR setting. Keywords: Information Extraction, Information Retrieval, Evaluation Methodologies 1. Introduction Tracking events are vital for situation awareness and de- cision making. The past two decades have witnessed an exponential growth of unstructured text, in which events are abundant. It is difficult, if not impossible, for a human user to keep up with the extremely high volume of events emerging every day. Event extraction comes to rescue. It aims at automatically extracting mentions of events 1 from text. However, pre- vious tasks, e.g., MUC (Grishman and Sundheim, 1996), ACE (Doddington et al., 2004b), and TAC-KBP (Freed- man and Gabbard, 2014; Mitamura et al., 2015), treat event extraction with "one size fits all" approaches, in which an event ontology needs to be defined a priori, and then event extractors are built just for the types in the ontology. These approaches often require extensive expertise from domain experts and linguists, in order to define event types rigor- ously 2. It will not work for personalized information needs that we frequently encounter nowadays. For instance, the ontology might be focused on socio-political events (e.g., Conflict and ProvideAid), but a user might be interested in events related to corporate finance (e.g., Merge and Acqui- sition), disease outbreaks, or sports. Furthermore, there are often disagreements among even experts on what they mean by certain event types (e.g., whether a verbal conflict counts as a Conflict event) and the level of granularity of interest (ProvideAid or more specifically, ProvideMilitaryAid). To satisfy the ever-increasing on-demand, personalized event-centric information needs, we introduce the few-shot 1 An event mention (EM) is an event with surrounding context (text), that are often triggered by a key word or phrase. An example is John attacked a bear. 2 For …

Content truncated to 500 words