WikiPossessions: Possession Timeline Generation as an Evaluation Benchmark for Machine Reading Comprehension of Long Texts

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1110–1117 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1110 WikiPossessions: Possession timeline generation as an evaluation benchmark for machine reading comprehension of long texts Dhivya Chinnappa†, Alexis Palmer∗, Eduardo Blanco∗ †Thomson Reuters dhivya.chinnappa@thomsonreuters.com ∗University of North Texas alexis.palmer@unt.edu, eduardo.blanco@unt.edu Abstract This paper presents WikiPossessions, a new benchmark corpus for the task of temporally-oriented possession (TOP), or tracking objects as they change hands over time. We annotate Wikipedia articles for 90 different well-known artifacts (paintings, diamonds, and archaeological artifacts), producing 799 artifact-possessor relations with associated attributes. For each article, we also produce a full possession timeline. The full version of the task combines straightforward entity-relation extraction with complex temporal reasoning, as well as verification of textual support for the relevant types of knowledge. Specifically, to complete the full TOP task for a given article, a system must do the following: a) identify possessors; b) anchor possessors to times/events; c) identify temporal relations between each temporal anchor and the possession relation it corresponds to; d) assign certainty scores to each possessor and each temporal relation; and e) assemble individual possession events into a global possession timeline. In addition to the corpus, we release evaluation scripts and a baseline model for the task. Keywords: possession, timeline, wikipossesions 1. Temporally-oriented possession Research in machine reading, text comprehension, and nat- ural language understanding continues to proceed at a rapid pace. Each time a new approach claims to surpass human performance on benchmark evaluation data sets, a new data set comes along, offering a new twist on the previous chal- lenges (Bowman et al., 2015; Rajpurkar et al., 2016; Dua et al., 2019, for example). At the same time, the community has questioned whether these data sets in fact require com- prehension of reading texts by the machine (Kaushik and Lipton, 2018). With this paper, we offer a new benchmark evaluation data set which combines several related tasks re- quiring different degrees of inferential complexity. Rather than focusing on question-answer pairs linked to particular text passages, the unit of analysis for this data set is a com- plete Wikipedia article. As a comparison, the average text length in our evaluation data set is an order of magnitude longer than the average passage in the DROP corpus (Dua et al., 2019), which was developed specifically to demand reasoning across longer text spans than previous reading comprehension benchmarks. The framework for the evaluation is the task of temporally- oriented possession (TOP), or tracking objects as they change hands over time. To complete the full TOP task for a given text, a system must do the following: a) identify all possessors of a target possessee; b) anchor possessors to times or events; c) determine temporal relations between each temporal anchor and the possession relation it corresponds to; d) label both possessors and temporal relations as certain or uncertain, according to whether the text provides explicit evidence; and Work done at …

Content truncated to 500 words