Improving Testsuites via Instrumentation

Improving Testsuites via Instrumentation Norbert BrSker Eschenweg 3 69231 Rauenberg Germany norbert, broeker@sap, com Abstract This paper explores the usefulness of a technique from software engineering, namely code instrumen- tation, for the development of large-scale natural language grammars. Information about the usage of grammar rules in test sentences is used to detect untested rules, redundant test sentences, and likely causes of overgeneration. Results show that less than half of a large-coverage grammar for German is actually tested by two large testsuites, and that i0- 30% of testing time is redundant. The methodology applied can be seen as a re-use of grammar writing knowledge for testsuite compilation. 1 Introduction Computational Linguistics (CL) has moved towards the marketplace: One finds programs employing CL- techniques in every software shop: Speech Recogni- tion, Grammar and Style Checking, and even Ma- chine Translation are available as products. While this demonstrates the applicability of the research done, it also calls for a rigorous development methodology of such CL application products. In this paper,lI describe the adaptation of a tech- nique from Software Engineering, namely code in- strumentation, to grammar development. Instru- mentation is based on the simple idea of marking any piece of code used in processing, and evaluating this usage information afterwards. The application I present here is the evaluation and improvement of grammar and testsuites; other applications are pos- sible. 1.1 Software Engineering vs. Grammar Engineering Both software and grammar development are simi- lar processes: They result in a system transforming some input into some output, based on a functional specification (e.g., cf. (Ciravegna et al., 1998) for the application of a particular software design method- ology to linguistic engineering). Although Grammar 1The work reported here was conducted during my time at the Institut fiir Maschinelle Sprachverarbeitung (IMS), Stuttgart University, Germany. Engineering usually is not based on concrete specifi- cations, research from linguistics provides an infor- mal specification. Software Engineering developed many methods to assess the quality of a program, ranging from static analysis of the program code to dynamic testing of the program's behavior. Here, we adapt dynamic testing, which means running the implemented pro- gram against a set of test cases. The test cases are designed to maximize the probability of detect- ing errors in the program, i.e., incorrect conditions, incompatible assumptions on subsequent branches, etc. (for overviews, cf. (Hetzel, 1988; Liggesmeyer, 1990)). 1.2 Instrumentation in Grammar Engineering How can we fruitfully apply the idea of measuring the coverage of a set of test cases to grammar devel- opment? I argue that by exploring the relation be- tween grammar and testsuite, one can improve both of them. Even the traditional usage of testsuites to indicate grammar gaps or overgeneration can profit from a precise indication of the grammar rules used to parse the sentences (cf. Sec.4). Conversely, one may use the grammar to improve the testsuite, both in terms of its coverage (cf. Sec.3.1) and its economy (cf. Sec.3.2). Viewed this way, testsuite writing can benefit from grammar development because both describe …

Content truncated to 500 words