Specifying a shallow grammatical representation for parsing purposes

Specifying a shallow grammatical representation for parsing purposes Atro Voutilainen and Timo J~irvinen Research Unit for Multilingual Language Technology P.O. Box 4 FIN-00014 University of Helsinki Finland At ro.Voutilainen,Timo. J arvinen@Helsinki.FI Abstract Is it possible to specify a grammatical representation (descriptors and their ap- plication guidelines) to such a degree that it can be consistently applied by different grammarians e.g. for producing a benchmark corpus for p.arser evalua- tion? Arguments for and against have been given, but very little empirical ev- idence. In this article we report on a double-blind experiment with a surface- oriented morphosyntactic grammatical representation used in a large-scale En- glish parser. We argue that a consis- tently applicable representation for mor- phology and also shallow syntax can be specified. A grammatical representation with a near-100% coverage of running text can be specified with a reasonable effort, especially if the representation is based on structural distinctions (i.e. it is structurally resolvable). 1 Introduction The central task of a parser is to assign gram- matical descriptions onto input sentences. Eval- uating a parser's output (as well as designing a computational lexicon and grammar) presupposes a predefined, parser-independent specification of the grammatical representation. Perhaps surprisingly, the possibility of specify- ing a workable grammatical representation is a matter of controversy, even at lower levels of anal- ysis, e.g. morphology (incl. parts of speech). Consider the following setting (the double-blind experiment). Two linguists trained to apply a tag set to running text according to application guide- lines (a "style sheet") are to analyse a given data individually. The results are then automatically compared, and the differences are jointly exam- ined by these linguists to see whether the differ- ences are due to inattention, or whether they are intentional (i.e. there is a genuine difference in analysis). - How many percentage points of all words in running text are retain a different anal- ysis after the differences due to inattention have been omitted? The higher this percentage, the more susceptible seems the possibility of specify- ing a workable grammatical representation. According to a pessimistic view (e.g. Church 1992), the part of speech of several percentage points of words in running text is impossible to agree on by different judges, even after negotia- tions. A more optimistic view can be found in (Leech and Eyes 1993, p. 39; Marcus et al. 1993, p. 328); they argue that a near-100% interjudge agreement is possible, provided the part-of-speech annotation is done carefully by experts. Unfortu- nately, they give very little empirical evidence for their position e.g. in terms of double-blind exper- iments. Supposing defining these lower levels of gram- matical representation is so problematic, the more distinctive levels should be even more difficult. If specifying the task of the parser - what the parser is supposed to do - turns out to be so problematic, one could even question the rationality of natu- ral language parser design as a whole. In other words, the controversy regarding the specifiability of a grammatical representation is a fundamental …

Content truncated to 500 words