Evaluating a Crosslinguistic Grammar Resource: A Case Study of Wambaya

Proceedings of ACL-08: HLT, pages 977–985, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Evaluating a Crosslinguistic Grammar Resource: A Case Study of Wambaya Emily M. Bender University of Washington Department of Linguistics Box 354340 Seattle WA 98195-4340 ebender@u.washington.edu Abstract This paper evaluates the LinGO Grammar Ma- trix, a cross-linguistic resource for the de- velopment of precision broad coverage gram- mars, by applying it to the Australian language Wambaya. Despite large typological differ- ences between Wambaya and the languages on which the development of the resource was based, the Grammar Matrix is found to pro- vide a significant jump-start in the creation of the grammar for Wambaya: With less than 5.5 person-weeks of development, the Wambaya grammar was able to assign correct seman- tic representations to 76% of the sentences in a naturally occurring text. While the work on Wambaya identified some areas of refine- ment for the Grammar Matrix, 59% of the Matrix-provided types were invoked in the fi- nal Wambaya grammar, and only 4% of the Matrix-provided types required modification. 1 Introduction Hand-built grammars are often dismissed as too ex- pensive to build on the one hand, and too brittle on the other. Nevertheless, they are key to various NLP applications, including those benefiting from deep natural language understanding (e.g., textual inference (Bobrow et al., 2007)), generation of well- formed output (e.g., natural language weather alert systems (Lareau and Wanner, 2007)) or both (as in machine translation (Oepen et al., 2007)). Of par- ticular interest here are applications concerning en- dangered languages: Endangered languages repre- sent a case of minimal linguistic resources, typically lacking even moderately-sized corpora, let alone treebanks. In the best case, one finds well-crafted descriptive grammars, bilingual dictionaries, and a handful of translated texts. The methods of pre- cision grammar engineering are well-suited to tak- ing advantage of such resources. At the same time, the applications of interest in the context of endan- gered languages emphasize linguistic precision: im- plemented grammars can be used to enrich existing linguistic documentation, to build grammar check- ers in the context of language standardization, and to create software language tutors in the context of language preservation efforts. The LinGO Grammar Matrix (Bender et al., 2002; Bender and Flickinger, 2005; Drellishak and Ben- der, 2005) is a toolkit for reducing the cost of creat- ing broad-coverage precision grammars by prepack- aging both a cross-linguistic core grammar and a series of libraries of analyses of cross-linguistically variable phenomena, such as major-constituent word order or question formation. The Grammar Ma- trix was developed initially on the basis of broad- coverage grammars for English (Flickinger, 2000) and Japanese (Siegel and Bender, 2002), and has since been extended and refined as it has been used in the development of broad-coverage grammars for Norwegian (Hellan and Haugereid, 2003), Modern Greek (Kordoni and Neu, 2005), and Spanish (Ma- rimon et al., 2007), as well as being applied to 42 other languages from a variety of language families in a classroom context (Bender, 2007). This …

Content truncated to 500 words