High Precision Treebanking—Blazing Useful Trees Using POS Information

Proceedings of the 43rd Annual Meeting of the ACL, pages 330–337, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics High Precision Treebanking — Blazing Useful Trees Using POS Information — Takaaki Tanaka,† Francis Bond,† Stephan Oepen,‡ Sanae Fujita† † {takaaki, bond, fujita}@cslab.kecl.ntt.co.jp ‡ oe@csli.stanford.edu † NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation ‡ Universitetet i Oslo and CSLI, Stanford Abstract In this paper we present a quantitative and qualitative analysis of annotation in the Hinoki treebank of Japanese, and in- vestigate a method of speeding annotation by using part-of-speech tags. The Hinoki treebank is a Redwoods-style treebank of Japanese dictionary definition sentences. 5,000 sentences are annotated by three dif- ferent annotators and the agreement evalu- ated. An average agreement of 65.4% was found using strict agreement, and 83.5% using labeled precision. Exploiting POS tags allowed the annotators to choose the best parse with 19.5% fewer decisions. 1 Introduction It is important for an annotated corpus that the mark- up is both correct and, in cases where variant anal- yses could be considered correct, consistent. Con- siderable research in the field of word sense disam- biguation has concentrated on showing that the an- notation of word senses can be done correctly and consistently, with the normal measure being inter- annotator agreement (e.g. Kilgariff and Rosenzweig, 2000). Surprisingly, few such studies have been car- ried out for syntactic annotation, with the notable exceptions of Brants et al. (2003, p 82) for the Ger- man NeGra Corpus and Civit et al. (2003) for the Spanish Cast3LB corpus. Even such valuable and widely used corpora as the Penn TreeBank have not been verified in this way. We are constructing the Hinoki treebank as part of a larger project in cognitive and computational lin- guistics ultimately aimed at natural language under- standing (Bond et al., 2004). In order to build the ini- tial syntactic and semantic models, we are treebank- ing the dictionary definition sentences of the most familiar 28,000 words of Japanese and building an ontology from the results. Arguably the most common method in building a treebank still is manual annotation, annotators (often linguistics students) marking up linguistic properties of words and phrases. In some semi-automated tree- bank efforts, annotators are aided by POS taggers or phrase-level chunkers, which can propose mark-up for manual confirmation, revision, or extension. As computational grammars and parsers have increased in coverage and accuracy, an alternate approach has become feasible, in which utterances are parsed and the annotator selects the best parse Carter (1997); Oepen et al. (2002) from the full analyses derived by the grammar. We adopted the latter approach. There were four main reasons. The first was that we wanted to de- velop a precise broad-coverage grammar in tandem with the treebank, as part of our research into natu- ral language understanding. Treebanking the output of the parser allows us to immediately identify prob- lems in the grammar, and improving the grammar directly improves the quality of the treebank in a mu- tually …

Content truncated to 500 words