Controlled Propagation of Concept Annotations in Textual Corpora

Controlled propagation of concept annotations in textual corpora Cyril Grouin LIMSI, CNRS, Universit´e Paris-Saclay, F-91405 Orsay cyril.grouin@limsi.fr Abstract In this paper, we presented the annotation propagation tool we designed to be used in conjunction with the BRAT rapid annotation tool. We designed two experiments to annotate a corpus of 60 files, first not using our tool, second using our propagation tool. We evaluated the annotation time and the quality of annotations. We shown that using the annotation propagation tool reduces by 31.7% the time spent to annotate the corpus with a better quality of results. Keywords: Annotation propagation; Corpus annotation; Texts 1. Introduction 1.1. Corpus annotation Corpus annotation is a crucial step to develop suitable natu- ral language processing (NLP) systems, to carry out evalu- ations of system outputs, or to train statistical models while using supervised machine-learning approaches (e.g., con- ditional random fields (Lafferty et al., 2001) for sequence labelling). Nevertheless, corpus annotation is a really time- consuming task. A useful way to reduce the time spent to annotate corpora consists in providing the annotators a pre-annotated version of the corpus to annotate. Automatic pre-annotations can be made through a lexicon mapping (i.e., all existing entities found in a lexicon would be automatically pre-annotated) or a system designed to annotate entities, either using a rule- based system or a machine-learning approach. The choice of the method used to pre-annotate corpora depends on the type of entity to process: regular entities such as numeric values can be formalized using rules while more complex entities or contextual annotations would be processed using statistical approaches. Annotators working on automatic pre-annotations have to check those annotations, in order to remove non relevant annotations and to complete miss- ing annotations. In a previous study, we demonstrated that using automatic pre-annotation based on CRF system both reduces the time spent by humans (annotators spent about 10% less time) and improves the quality of the final anno- tations (we computed a gain of 6 points in κ inter-annotator agreements), in comparison with annotation task made on similar raw corpora (Grouin and N´ev´eol, 2014). Another solution to reduce annotation time consists in se- lecting documents from a corpus, parts of documents, or parts of text (e.g., a few sentences), using a sampling pro- cess (Patton and Potok, 2006; Kantner et al., 2011), in or- der to annotate only a few samples of corpora. Those sam- ples are considered by the corpus manager as representative enough of phenomenon to annotate and study. 1.2. Annotation propagation The basic principle of annotation propagation relies on ex- isting annotations that will be associated with new docu- ments, for which parts—either a single word or a whole part depending on the type of annotation to be made—are found to be similar with previously annotated documents. Two main objectives are expected while using annotation propagation systems: first, the reduction of time spent by humans to annotate corpora, and second, an improvement of the final quality of annotations made. As a consequence, human annotators …

Content truncated to 500 words