A Web-based System for Crowd-in-the-Loop Dependency Treebanking

A Web-based System for Crowd-in-the-Loop Dependency Treebanking Stephen Tratz∗, Nhien Phan† ∗U.S. Army Research Laboratory, Adelphi, Maryland, USA †University of Maryland, College Park, Maryland, USA stephen.c.tratz.civ@mail.mil, nphan12@terpmail.umd.edu Abstract Treebanks exist for many different languages, but they are often quite limited in terms of size, genre, and topic coverage. It is difficult to expand these treebanks or to develop new ones in part because manual annotation is time-consuming and expensive. Human-in-the-loop methods that leverage machine learning algorithms during the annotation process are one set of techniques that could be employed to accelerate annotation of large numbers of sentences. Additionally, crowdsourcing could be used to hire a large number of annotators at relatively low cost. Currently, there are few treebanking tools available that support either human-in-the-loop methods or crowdsourcing. To address this, we introduce CROWDTREE, a web-based interactive tool for editing dependency trees. In addition to the visual frontend, the system has a Java servlet that can train a parsing model during the annotation process. This parsing model can then be applied to sentences as they are requested by annotators so that, instead of annotating sentences from scratch, annotators need only to edit the model’s predictions, potentially resulting in significant time savings. Multiple annotators can work simultaneously, and the system is even designed to be compatible with Mechanical Turk. Thus, CROWDTREE supports not simply human-in-the-loop treebanking, but crowd-in-the-loop treebanking. Keywords: annotation, dependency trees, visualization, GUI, crowdsourcing, treebanking 1. Introduction Although treebanks exist for a number of languages, they are often quite small in size. Even some of the largest and most useful treebanks are restricted to only a hand- ful of sources, genres, and/or topic areas—the venerable Penn Treebank (Marcus et al., 1993) being a prime ex- ample of this. Unfortunately, manual annotation is expen- sive and slow, which impedes both the expansion of exist- ing treebanks and the creation of new ones. For example, The Prague Dependency Treebank, which contains over 1 million syntactically connected words, cost approximately $600,000 to create (B¨ohmov´a et al., 2003). One method for accelerating the treebanking process is to utilize human- in-the-loop1 methods that leverage machine learning algo- rithms during annotation. Such algorithms could pre-parse the data, highlight attachment decisions of greater impor- tance, and/or suggest sentences for annotation that are most likely to prove useful as training examples. Another useful method is crowdsourcing, which taps into a large (and typ- ically inexpensive) worker pool. Together, these comple- mentary techniques hold tremendous promise for reducing the time and cost of creating large datasets for natural lan- guage processing, including treebanks. In order to facilitate faster and less expensive dependency tree annotation, this paper presents CROWDTREE, which is, to the best of our knowledge, the first web-based treebanking tool designed specifically for human-in-the-loop, and even crowd-in-the- loop, tree annotation. The system consists of two primary components, an inter- active graphical frontend, shown in Figure 1, and a Java servlet. The frontend, which enables users to visually con- struct/edit dependency trees using drag-and-drop, is based on the EASYTREE …

Content truncated to 500 words