Dimensions of Deep Grammar Validation

Proceedings of the Australasian Language Technology Workshop 2005, pages 1–3, Sydney, Australia, December 2005. Dimensions of Deep Grammar Validation Dan Flickinger Oslo, Saarland, and Stanford Universities danf@csli.stanford.edu Abstract In order to arrive at a more disciplined ap- proach to the sustained development of linguis- tically rich grammars, I present a methodology for grammar validation, identifying principal di- mensions of the task, and illustrating the appli- cation of the method for one release cycle of the open-source English Resource Grammar. 1 Introduction Broad-coverage grammars for natural language processing which encode rich linguistic de- scription are resources which develop relatively slowly, through many iterations of modification and testing, and ideally through exposure to the demands of a variety of NLP tasks. As with most large software components, such gram- mars are designed to exhibit a complexity of behavior which presents serious challenges for quality assurance from one release of a gram- mar to the next release. There is by now a substantial literature on many aspects of the evaluation of NLP software, including gram- mars, much of it focused around ’black box’ or functional evaluation within some specific task domain(Hirschman and Thompson, 1998), (Sparck Jones and Galliers, 1998). Methods and tools have also been developed for ’glass box’ or structural evaluation (EAGLES, 1996), with one line of work analyzing the internal formal coherence of such deep grammars, for example to flag inconsistencies or identify unused rules or constraints in the grammar code (Broeker, 2000), (Barr and Siefring, 2004), and another line of work using paraphrase generation to il- luminate properties of a bi-directional grammar not easily detected when parsing (Dymetman and Isabelle, 1988). And finally, there has been work on developing a methodology of sustained grammar development, whereby these labor- intensive, long-lived resources undergo periodic modification intended to improve corpus cover- age, or processing efficiency, or linguistic preci- sion, or more typically all three at once (Oepen and Flickinger, 1998). Such a methodology must incorporate a set of disciplined procedures for validation of the resulting grammar, ensur- ing that it meets the expectations of its engi- neers, and explicitly communicates the interface specifications for its use in applications. In this talk I describe an instantiation of a method for natural language grammar valida- tion (cf. (Barr and Klavans, 2001)) to iden- tify and motivate the multiple dimensions of the procedure, and to illustrate how many of the the tools and techniques proposed in the NLP evaluation literature are being exploited by en- gineers of large deep grammars. For concrete- ness, I present the method used in maintaining and extending the English Resource Grammar (ERG (Flickinger, 2000), a semantically pre- cise, broad-coverage Head-driven Phrase Struc- ture Grammar (HPSG) implementation used for both parsing and generation in several NLP applications, being developed within the Deep Linguistic Processing with HPSG Initia- tive (DELPH-IN: www.delph-in.net). At the heart of this method is the use of a growing set of test suites of two kinds, to- gether with a sophisticated grammar profiling tool, [incr tsdb()], which collects …

Content truncated to 500 words