Detecting Annotation Scheme Variation in Out-of-Domain Treebanks

Detecting Annotation Scheme Variation in Out-of-Domain Treebanks Yannick Versley, Julius Steen ICL Univ. Heidelberg / Leibniz ScienceCampus (versley|steen)@cl.uni-heidelberg.de Abstract To ensure portability of NLP systems across multiple domains, existing treebanks are often extended by adding trees from interesting domains that were not part of the initial annotation effort. In this paper, we will argue that it is both useful from an application viewpoint and enlightening from a linguistic viewpoint to detect and reduce divergence in annotation schemes between extant and new parts in a set of treebanks that is to be used in evaluation experiments. The results of our correction and harmonization efforts will be made available to the public as a test suite for the evaluation of constituent parsing. Keywords: treebank, variation, annotation schemes 1. Introduction In treebank creation, there is a growing trend to complement existing in-domain (newspaper text) tree- banks by out-of-domain treebanks that allow re- searchers to study syntactic variation and train statis- tical parsing models on texts of other domains. Most often, the creation of these out-of-domain tree- banks follows the existing annotation guidelines cre- ated by the creators of the newspaper treebank, which would – at least in principle – make it possible to eval- uate the generalization performance both of purely- supervised and domain-adapted parsers on these new datasets. However, some caution is in order, as in many cases, annotation guidelines or even informal practices guiding the annotation deviate.In the case of Dredze et al. (2007), researchers report that the diver- gence between the annotation schemes of Penn Tree- bank and the Penn Biomedical treebank was greater than the influence of divergence between domains, and that manual annotation conversion of the training dataset without any domain adaptation yielded better results than those of many other participants’. It has long been argued that detecting and correcting errors in treebanks is a sensible thing to do, as these become a more prominent source of errors as the per- formance of automatic taggers and parsers gets bet- ter (Blaheta, 2002; Manning, 2011). In a study on part-of-speech tag usage in treebanking (Telljohann et al., 2013), it was found that even after the first pub- lic release of a treebank, about 0.8% of tags would be changed in routine revisions (which would correspond about to one every eight sentences). In the case of out- of-domain treebanks, it is often the case that a new group of people is involved in the construction, and that the annotations between old and new resource dif- fer in tacit, and sometimes also explicit, aspects of the annotation scheme. Dredze et al. (2007) argue that, in adapting parsers from the Penn Treebank to the Penn Biomedical Treebank, the primary cause of loss from adaptation is from differences in the annotation guide- lines themselves. While the OntoNotes version of the Penn Treebank (Weischedel et al., 2008) now allows to carry out this kind of domain adaptation experiment, it is to be ex- pected that any such experiment with domain adapta- tion will yield …

Content truncated to 500 words