Ellogon Casual Annotation Infrastructure

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3360–3365 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3360 Ellogon Casual Annotation Infrastructure Georgios Petasis, Leonidas Tsekouras Institute of Informatics and Telecommunications, National Centre for Scientific Research (N.C.S.R.) “Demokritos”, P.O. BOX 60228, Aghia Paraskevi, GR-153 10, Athens, Greece. {petasis, ltsekouras}@iit.demokritos.gr Abstract This paper presents a new annotation paradigm, casual annotation, along with a proposed architecture and a reference implementation, the Ellogon Casual Annotation Tool, which implements this paradigm and architecture. The novel aspects of the proposed paradigm originate from the vision to tightly integrate annotation with the casual, everyday activities of users. Annotating in a less “controlled” environment, and removing the bottleneck of selecting content and importing it to annotation infrastructures, casual annotation provides the ability to vastly increase the content that can be annotated and ease the annotation process through automatic pre-training. The proposed paradigm, architecture and reference implementation has been evaluated for more than two years on an annotation task related to sentiment analysis. Evaluation results suggest that, at least for this annotation task, there is a huge improvement in productivity after casual annotation adoption, in comparison to the more traditional annotation paradigms followed in the early stages of the annotation task. Keywords: Authoring Tools, Corpus (Creation, Annotation, etc.), LR Infrastructures and Architectures, Casual Annotation, An- notation Tools, Collaborative Annotation, Web-based annotation tools 1. Introduction The development and maintenance of annotated corpora can be significantly facilitated through the use of annota- tion tools, as annotation tools can control most aspects of the annotation process, from the presentation of the relevant information to the annotators to the validation of annotated information according to a predefined schema. A plethora of annotation tools has been presented during the last two decades (Uren et al., 2006; Fragkou et al., 2008a; Katakis et al., 2016), covering a wide range of annotation tasks and offering various levels of support. Annotation solutions can be divided into manual and semi-automatic methods: man- ual solutions provide the required infrastructure (i.e. stor- age management, graphical user interface, etc.) for anno- tators to annotate a corpus with a completely manual ap- proach, where all information must be manually entered by the annotators. Semi-automatic solutions on the other hand, try to pre-annotate corpora, reducing the role of an- notators into validation of existing pre-annotation. How- ever, several of the existing annotation tools are desktop applications, allowing the annotation of corpora found on a single computer. A more recent category of annotation solutions, are distributed or collaborative annotation tools, where several annotators (not necessarily co-located) can annotate the same corpus, and in some cases even the same document. However, the construction of annotation tools that operate in a distributed environment is a challenging task, while the majority of these tools are implemented as Web applications (such as Ellogon’s Collaborative Annota- tion Tool1 (Katakis et al., 2016), WebAnno2 (Yimam et al., 2013) or BRAT3 (Stenetorp et al., 2012a)), having to cope with the …

Content truncated to 500 words