SemEval-2018 Task 5: Counting Events and Participants in the Long Tail

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 70–80 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics SemEval-2018 Task 5: Counting Events and Participants in the Long Tail Marten Postma, Filip Ilievski, Piek Vossen Vrije Universiteit Amsterdam The Netherlands {m.c.postma, f.ilievski, piek.vossen}@vu.nl Abstract This paper discusses SemEval-2018 Task 5: a referential quantification task of counting events and participants in local, long-tail news documents with high ambiguity. The com- plexity of this task challenges systems to es- tablish the meaning, reference and identity across documents. The task consists of three subtasks and spans across three domains. We detail the design of this referential quantifica- tion task, describe the participating systems, and present additional analysis to gain deeper insight into their performance. 1 Introduction We present a “referential quantification” task that requires systems to establish the meaning, refer- ence and identity of events1 and participants in news articles. By “referential quantification”, we mean questions concerning the number of inci- dents of an event type (e.g. How many killing in- cidents happened in 2016 in Columbus, MS?) or participants in roles (e.g. How many people were killed in 2016 in Columbus, MS?), as opposed to factoid questions for specific properties of indi- vidual events and entities (e.g. When was 2pac murdered?). The questions are given with cer- tain constraints on the location, time, participants, and event types, which requires understanding of the meaning of words mentioning these properties (e.g. Word Sense Disambiguation), but also ad- equately establishing the identity (e.g. reference and coreference) across mentions. The task thus represents both an intrinsic and application-based evaluation, as systems are forced to resolve ambi- guity of meaning and reference, as well as varia- tion in reference in order to answer the questions. 1By event, we denote a specific instance of an event, e.g. a killing incident happening at a specific location, time, and involving certain participants. Figure 1 shows an overview of our quantifica- tion task. We provide the participants with a set of questions and their corresponding news docu- ments.2 Systems are asked to distill event- and participant-based knowledge from the documents to answer the question. Systems submit both a nu- meric answer (3 events in Figure 1), and the corre- sponding events with their mentions found in the provided texts (e.g., the leftmost incident in Fig- ure 1 is referred to by the coreferring mentions “killed” and “assault” found in two separate doc- uments). Systems are evaluated on both the nu- meric answers as well as on the sets of coreferring mentions. Mentions are represented by tokens and offsets provided by the organizers. The incidents and their corresponding news arti- cles are obtained from structured databases, which greatly reduces the need for annotation and mainly requires validation instead. Given this data and using a metric-driven strategy, we created a task that further maximizes ambiguity and variation of the data in relation to the questions. This ambigu- ity and variation includes a substantial amount of low-frequent, local events …

Content truncated to 500 words