Resource Interoperability for Sustainable Benchmarking: The Case of Events

Resource Interoperability for Sustainable Benchmarking: The Case of Events Chantal van Son, Oana Inel, Roser Morante, Lora Aroyo, Piek Vossen Vrije Universiteit Amsterdam De Boelelaan 1105, 1081 HV Amsterdam, The Netherlands {c.m.van.son, oana.inel, r.morantevallejo, lora.aroyo, piek.vossen}@vu.nl Abstract With the continuous growth of benchmark corpora, which often annotate the same documents, there is a range of opportunities to compare and combine similar and complementary annotations. However, these opportunities are hampered by a wide range of problems that are related to the lack of resource interoperability. In this paper, we illustrate these problems by assessing aspects of interoperability at the document-level across a set of 20 corpora annotated with (aspects of) events. The issues range from applying different document naming conventions, to mismatches in textual content and structural/conceptual differences among annotation schemes. We provide insight into the exact document intersections between the corpora by mapping their document identifiers and perform an empirical analysis of event annotations showing their compatibility and consistency in and across the corpora. This way, we aim to make the community more aware of the challenges and opportunities and to inspire working collaboratively towards interoperable resources. Keywords: resource interoperability, annotation consistency, events 1 Introduction Language resources are at the core of most research in computational linguistics and natural language processing (NLP) for system development and benchmarking. There is already a wealth of resources available and the collec- tion is continuously expanding. With this growth, moni- toring their creation and making them interoperable is be- coming increasingly important for finding, accessing and reusing existing corpora. For example, different annotation layers applied to the same document provide valuable op- portunities for studying and leveraging interdependencies between different types of annotation, but these opportuni- ties are hampered when too laborious conversion steps are required to resolve structural or conceptual differences in their representations (Chiarcos, 2012a). Despite the efforts of many initiatives working towards solutions (Section 2), interoperability issues still persist today and become more and more problematic. On the one hand, this is because reaching consensus on standards and best practices is not a straightforward task and naturally takes time. On the other hand, we hypothesize that the problems that arise when in- teroperability is lacking have not been illustrated enough for the necessity of solutions to be widely recognized. Although there are various types of language resources, we limit our discussion to annotated text corpora for which we define interoperability at the levels presented in Figure 1. At the corpus-level, interoperability involves documenting and representing metadata of the data collection as a whole such as its name, language, type, genre, source, creator, year, size, etc., to enable resolving the identity of corpora in a uniform way. At the document-level, we can further distinguish between the metadata of the document (e.g. file- name, language, size) and its body, where the latter consists of a textual string and the linguistic annotations of its sub- strings (e.g. sentences, phrases, tokens). With respect to the annotations, we adopt the distinction between structural interoperability (annotations of …

Content truncated to 500 words