Do you Feel Certain about your Annotation? A Web-based Semantic Frame Annotation Tool Considering Annotators’ Concerns and Behaviors

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 7132–7139 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 7132 Do you Feel Certain about your Annotation? A Web-based Semantic Frame Annotation Tool Considering Annotators’ Concerns and Behaviors Regina Stodden & Behrang QasemiZadeh & Laura Kallmeyer Heinrich Heine University, D¨usseldorf, Germany {stodden, zadeh, kallmeyer}@phil.hhu.de Abstract In this paper, we present an open-source web-based application with a responsive design for modular semantic frame annotation (SFA). Besides letting experienced and inexperienced users perform suggestion-based and slightly-controlled annotations, the system keeps track of the time and changes annotators made during the annotation process and logs certain metadata. This collected metadata can be used to get new insights regarding the difficulty of annotating specific types of frames, and as an input of an annotation cost measurement for an active learning algorithm. The tool was already used to build a manually annotated corpus with semantic frames and its arguments for task 2 of SemEval 2019 regarding unsupervised lexical frame induction (QasemiZadeh et al., 2019). Although English sentences from the Wall Street Journal corpus of the Penn Treebank (Marcus et al., 1999) are annotated for this task, it is also possible to use the proposed tool for the annotation of sentences in any other languages. Keywords: annotation tool, semantic frames, multilingual semantic annotation tool 1. Introduction In computational linguistics, manually annotated corpora are in high demand. In machine-learning-based natural lan- guage processing tasks, corpora with manual annotations are necessary to train, evaluate, and compare systems and algorithms in terms of quantitative measures. However, the development of manually annotated corpora is resource- intensive and complex; and, often experienced annotators and annotation tools specialized for the purpose of the an- notation task are required. This paper addresses these prob- lems in the context of semantic frame annotation by intro- ducing a web-based open source software. Simply put, semantic frames as used in this paper are event representations that are assigned to lexical units. A frame consists of a frame type (also known as event type) and a set of semantic roles (slots/elements), each of which links the item evoking the event to the lexical filler of the role (see Fig. 1 for an example)1. Frame information is useful for various natural language understanding tasks, e.g., question answering, information extraction, or text summarization. The basic idea is that the understanding of an utterance re- quires knowledge about the denoted events and the roles of their participants (Fillmore and Baker, 2010). The sentence (1) for instance expresses an event of ”something originat- ing from somewhere”. The verbal multi-word expression (VMWE) come from evokes the ORIGIN frame with the se- mantic roles or frame elements ENTITY and ORIGIN. The two role labels are assigned to the lexical units Criticism and Wall Street (see Fig. 1). (1) Criticism of futures COMES FROM Wall Street. FrameNet (Ruppenhofer et al., 2016), the most well-known 1Throughout the paper, frame types are highlighted in red, and frame elements, …

Content truncated to 500 words