Apportioning Development Effort in a Probabilistic LR Parsing System Through Evaluation

APPORTIONING DEVELOPMENT EFFORT IN A PROBABILISTIC LR PARSING SYSTEM THROUGH EVALUATION John Carroll Ted Briscoe Cognitive and Computing Sciences Computer Laboratory University of Sussex University of Cambridge Brighton BN1 9QH, UK Pembroke Street, Cambridge CB2 3QG, UK john. carroll@cogs.susx, ac. uk ejb @cl. cam. ac. uk Abstract We describe an implemented system for robust domain-independent syntactic parsing of English, using a unification-based grammar of part-of- speech and punctuation labels coupled with a probabilistic LR parser. We present evaluations of the system's performance along several differ- ent dimensions; these enable us to assess the con- tribution that each individual part is making to the success of the system as a whole, and thus prioritise the effort to be devoted to its further enhancement. Currently, the system is able to parse around 80% of sentences in a substantial corpus of general text containing a number of distinct genres. On a random sample of 250 such sentences the system has a mean crossing bracket rate of 0.71 and recall and precision of 83% and 84~0 respectively when evaluated against manually-disambiguated analyses I . 1. INTRODUCTION This work is part of an effort to develop a ro- bust, domain-independent syntactic parser capa- ble of yielding the unique correct analysis for un- restricted naturally-occurring input. Our goal is to develop a system with performance compara- ble to extant part-of-speech taggers, returning a syntactic analysis from which predicate-argument structure can be recovered, and which can sup- port semantic interpretation. The requirement for a domain-independent analyser favours statistical 1Some of this work was carried out while the second author was visiting Rank Xerox, Grenoble. The work was also supported by UK DTI/SALT project 41/5808 'Integrated Language Database', and by SERC/EPSRC Advanced Fellowships to both au- thors. Geoff Nunberg provided encouragement and much advice on the analysis of punctuation, and Greg Grefenstette undertook the original corpus tokenisa- tion and segmentation for the punctuation experi- ments. Bernie .]ones and Kiku Ribas made helpful comments on an earlier draft. We are responsible for any mistakes. 92 techniques to resolve ambiguities, whilst the lat- ter goal favours a more sophisticated grammatical formalism than is typical in statistical approaches to robust analysis of corpus material. Briscoe ~ Carroll (1993) describe a proba- blistic parser using a wide-coverage unification- based grammar of English written in the Alvey Natural Language Tools (ANLT) metagrammat- ical formalism (Briscoe et al., 1987), generating around 800 rules in a syntactic variant of the Def- inite Clause Grammar formalism (DCG, Pereira Warren, 1980) extended with iterative (Kleene) operators. The ANLT grammar is linked to a lex- icon containing about 64K entries for 40K lex- emes, including detailed subcategorisation infor- mation appropriate for the grammar, built semi- automatically from a learners' dictionary (Car- roll L= Grover, 1989). The resulting parser is efficient, constructing a parse forest in roughly quadratic time (empirically), and efficiently re- turning the ranked n-most likely analyses (Car- roll, 1993, 1994). The probabilistic model is a refinement of probabilistic context-free grammar (PCFG) conditioning CF 'backbone' rule applica- …

Content truncated to 500 words