Interannotator Agreement for Lexico-Semantic Annotation of a Corpus

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1842–1848 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1842 Interannotator Agreement for Lexico-Semantic Annotation of a Corpus Elżbieta Hajnicz Institute of Computer Science, Polish Academy of Sciences ul. Jana Kazimierza 5, 01-248 Warszawa, Poland hajnicz@ipipan.waw.pl Abstract This paper examines the procedure for lexico-semantic annotation of the Basic Corpus of Polish Metaphors that is the first step for annotating metaphoric expressions occurring in it. The procedure involves correcting the morphosyntactic annotation of part of the corpus that is automatically annotated on the morphosyntactic level. The main procedure concerns annotation of adjectives, adverbs, nouns and verbs (including gerunds and participles), including abbreviations of the words that belong to the above classes. It is composed of three steps: deciding whether a particular occurrence of a word is asemantic (e.g. anaphoric or strictly grammatical), whether we are dealing with a multi-word expression, reciprocal usages of the się marker and pluralia tantum, which may involve annotation with two lexical units (having two different lemmas) for a single token. We propose an interannotator agreement statistics adequate for this procedure. Finally, we discuss the preliminary results of annotation of a fragment of the corpus. Keywords: corpus, lexico-semantic annotation, interannotator agreement, Polish 1. Introduction In this paper we want to describe the procedure for lexico- semantic annotation of the Basic Corpus of Polish Metaphor (BCPM), which is a first step for annotation of metaphoric expressions. This task is part of the Cognitive and socio- cultural analysis of metaphoric expressions in Polish texts project, aimed at, among other things, automatic detection of metaphoric expressions in Polish texts. Annotated corpora form a basis for natural language pro- cessing. Several corpora annotated on various levels of lin- guistic information exist: morphosyntactic, syntactic (shal- low or deep), multi-word expression (MWEs), and finally semantic, including lexico-semantic, and for various lan- guages. The usual procedure is to manually annotate a small cor- pus, use it to train NLP tools and annotate the whole corpus (a substantially larger one) by means of those tools. The quality of the manual annotation is crucial here. To ensure this happens, at least two linguists usually annotate each text sample in the corpus and the conflicts are resolved by a superannotator. Furthermore, a so-called interannotator agreement is calculated to show the difficulty of the task and the quality of its performance. Several statistics are used to calculate the interannotator agreement. The simplest calculate the percentage of iden- tical annotation. The most popular coefficients are Cohen’s (Cohen, 1960) κ, Scott’s (Scott, 1955) π and Bennett’s S (Bennett et al., 1954) which take into account the possibility of chance agreement. All the statistics satisfy the equation (1), but they differ in calculating P(E). S is the simplest in this respect, as it considers only the cardinality of the set of classes, whereas the two other coefficients take into ac- count the distribution of decisions of annotators in the cor- pus ˆPa1(k), ˆPa1(k), …

Content truncated to 500 words