The Royal Society Corpus 6.0: Providing 300+ Years of Scientific Writing for Humanistic Study

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 74–79 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 74 Model-based annotation of coreference Rahul Aralikatte and Anders Søgaard University of Copenhagen {rahul, soegaard}@di.ku.dk Abstract Humans do not make inferences over texts, but over models of what texts are about. When annotators are asked to annotate coreferent spans of text, it is therefore a somewhat unnatural task. This paper presents an alternative in which we preprocess documents, linking entities to a knowledge base, and turn the coreference annotation task – in our case limited to pronouns – into an annotation task where annotators are asked to assign pronouns to entities. Model-based annotation is shown to lead to faster annotation and higher inter-annotator agreement, and we argue that it also opens up for an alternative approach to coreference resolution. We present two new coreference benchmark datasets, for English Wikipedia and English teacher-student dialogues, and evaluate state-of-the-art coreference resolvers on them. Keywords: Coreference resolution, Linguistic mental models 1. Introduction Language comprehension is often seen as the incremental update of a mental model of the situation described in the text (Bower and Morrow, 1990). The model is incremen- tally updated to represent the contents of the linguistic input processed so far, word-by-word or sentence-by-sentence. In this paper, we restrict ourselves to one central feature shared by most theories of mental models: they include a list of entities previously introduced in the text. This corre- sponds to the constants of first-order models or the referents associated with different roles in frame semantics. By mod- els we thus simply mean a set of entities. Obviously, this is not sufficient to represent the meaning of texts, but fo- cusing exclusively on annotating nominal coreference, we can ignore relations and predicates for this work. We will use the term model-based annotation to refer to linguistic annotation using model representations to bias or ease the work of the annotators. Mental models have previously been discussed in linguis- tics literature on coreference (Runner et al., 2003). The motivation has often been that some pronouns refer to en- tities that are not explicitly mentioned in the previous text, but are supposedly available in the reader’s mental model of the text, by inference. Consider, for example: (1) I knocked on the door of room 624. He wasn’t in. The introduction of the referent of he in (1) is implied by the introduction of the entity room 624. In this paper, we present a new approach to annotating coreference that en- ables simple annotation of examples such as (1): Instead of asking an annotator to relate pronouns and previous spans of text, we ask the annotator to link pronouns and entities in document models. Moreover, we argue that model-based annotation reduces the cognitive load of annotators, which we experimentally test by comparing inter-annotator agree- ment and annotator efficiency across comparable annota- tion experiments. Fig. 1 showcases a concrete example from the collected dataset. Contributions …

Content truncated to 500 words