Ad Hoc Treebank Structures

Proceedings of ACL-08: HLT, pages 362–370, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Ad Hoc Treebank Structures Markus Dickinson Department of Linguistics Indiana University md7@indiana.edu Abstract We outline the problem of ad hoc rules in treebanks, rules used for specific construc- tions in one data set and unlikely to be used again. These include ungeneralizable rules, erroneous rules, rules for ungrammatical text, and rules which are not consistent with the rest of the annotation scheme. Based on a sim- ple notion of rule equivalence and on the idea of finding rules unlike any others, we develop two methods for detecting ad hoc rules in flat treebanks and show they are successful in de- tecting such rules. This is done by examin- ing evidence across the grammar and without making any reference to context. 1 Introduction and Motivation When extracting rules from constituency-based tree- banks employing flat structures, grammars often limit the set of rules (e.g., Charniak, 1996), due to the large number of rules (Krotov et al., 1998) and “leaky” rules that can lead to mis-analysis (Foth and Menzel, 2006). Although frequency-based cri- teria are often used, these are not without problems because low-frequency rules can be valid and po- tentially useful rules (see, e.g., Daelemans et al., 1999), and high-frequency rules can be erroneous (see., e.g., Dickinson and Meurers, 2005). A key issue in determining the rule set is rule generaliz- ability: will these rules be needed to analyze new data? This issue is of even more importance when considering the task of porting a parser trained on one genre to another genre (e.g., Gildea, 2001). In- frequent rules in one genre may be quite frequent in another (Sekine, 1997) and their frequency may be unrelated to their usefulness for parsing (Foth and Menzel, 2006). Thus, we need to carefully consider the applicability of rules in a treebank to new text. Specifically, we need to examine ad hoc rules, rules used for particular constructions specific to one data set and unlikely to be used on new data. This is why low-frequency rules often do not extend to new data: if they were only used once, it was likely for a specific reason, not something we would expect to see again. Ungeneralizable rules, however, do not extend to new text for a variety of reasons, not all of which can be captured strictly by frequency. While there are simply phenomena which, for var- ious reasons, are rarely used (e.g., long coordinated lists), other ungeneralizable phenomena are poten- tially more troubling. For example, when ungram- matical or non-standard text is used, treebanks em- ploy rules to cover it, but do not usually indicate un- grammaticality in the annotation. These rules are only to be used in certain situations, e.g., for ty- pographical conventions such as footnotes, and the fact that the situation is irregular would be useful to know if the purpose of an induced grammar is to support robust parsing. And these rules are out- right damaging …

Content truncated to 500 words