A Distributed Architecture for Interactive Parse Annotation

Proceedings of the Australasian Language Technology Workshop 2005, pages 207–214, Sydney, Australia, December 2005. A Distributed Architecture for Interactive Parse Annotation Baden Hughes Department of Computer Science and Software Engineering The University of Melbourne Victoria 3010, Australia badenh@csse.unimelb.edu.au James Haggerty, Saritha Manickam Joel Nothman, James R. Curran School of Information Technologies University of Sydney NSW 2006, Australia {j.h,jnothman,saritha}@student.usyd.edu.au james@it.usyd.edu.au Abstract In this paper we describe a modular system ar- chitecture for distributed parse annotation us- ing interactive correction. This involves inter- actively adding constraints to an existing parse until the returned parse is correct. Using a mixed initiative approach, human annotators in- teract live with distributed ccg parser servers through an annotation gui. The examples pre- sented to each annotator are selected by an ac- tive learning framework to maximise the value of the annotated corpus for machine learners. We report on an initial implementation based on a distributed workflow architecture. 1 Introduction Annotating sentences with parse trees is per- haps the most complex and intensive linguistic annotation. The time and expense of develop- ing parsed corpora is almost prohibitive. As a result there are only a small number of such cor- pora, including the Penn Treebank (Marcus et al., 1994), the German TiGer Corpus (Skut et al., 1997) and more recently the LinGO Red- woods Treebank (Oepen et al., 2002). These corpora are also limited in size, typically around one million words. Unfortunately, the statistical approaches to parsing which have been most successful rely heavily on both the quality and quantity of an- notated resources. Also, these approaches are very sensitive to the statistical properties of the corpus, and so a parser trained on one genre may perform badly on another (Gildea, 2001). Another major problem with parsed corpora is that they must, at least to some extent, fol- low a particular syntactic theory or formalism. This is a major difficulty for two reasons: firstly, it means we need separate annotated corpora for each formalism; and secondly, it means that comparing parser evaluations across formalisms is difficult. Fully automated conversion of parse trees be- tween formalisms is difficult because each analy- ses certain constructs in idiosyncratic ways. An example is CCGbank (Hockenmaier and Steed- man, 2002), a treebank of Combinatory Cate- gorial Grammar (Steedman, 2000) derivations which were converted semi-automatically from the Penn Treebank trees. The result still re- quired laborious editing to produce idiomatic ccg derivations (Hockenmaier, 2003). We intend to create a new corpus of ccg derivations on a wide range of text. We face three key problems: 1) selecting sentence to an- notate which creates the most useful corpus for statistical parsers. 2) maximising the annota- tor efficiency and minimising error; 3) allowing distributed annotators to share expertise. The selection problem is addressed using ac- tive learning (al). Active learning involves computing which training instances provide the most new information to one (or more) machine learners (Cohn et al., 1995; Dagan and Engel- son, 1995). The annotators become oracles an- swering specific queries posed by the …

Content truncated to 500 words