Multi-Site Data Collection for a Spoken Language Corpus

Multi-Site Data Collection for a Spoken Language Corpus MADCOW * Contact: Lynette Hirschman NE43-643 Spoken Language Systems Group MIT Laboratory for Computer Science, Cambridge, MA 02139. e-maih lynette~goldilocks.lcs.mit.edu ABSTRACT This paper describes a recently collected spoken language cor- pus for the ATIS (Air Travel Information System) domain. This data collection effort has been co-ordinated by MAD- COW (Multi-site ATIS Data COllection Working group). We summarize the motivation for this effort, the goals, the imple- mentation of a multi-site data collection paradigm, and the accomplishments of MADCOW in monitoring the collection and distribution of 12,000 utterances of spontaneous speech from five sites for use in a multi-site common evaluation of speech, natural language and spoken language. 1. Introduction Following the February 1991 DARPA Speech and Natu- ral Language Workshop, the DARPA Spoken Language contractors decided to institute a multi-site data collec- tion paradigm in order to: • support a common evaluation on speech, natural language and spoken language; • maximize the amount of data collected; • provide some diversity in data collection paradigms; • reduce cost to any one site by sharing the data col- lection activity across multiple participating sites. To co-ordinate this effort, MADCOW was formed in May 1991 with a representative from each of the participat- ing sites. This included the six sites planning to collect and evaluate on the data: AT&T, BBN, CMU, MIT, SRI and Paramax (formerly Unisys), it included NIST, which was responsible for data validation, distribution and se- lection and scoring of test material, and it included the Annotation group at SRI, responsible under a separate contract for annotating the data with database reference answers. *This paper was written under the auspices of the Multi- Site ATIS Data Collection Working group (MADCOW) by L. Hirschman, M. Bates, D. Dahl, W. Fisher, J. Garofolo, K. Hunicke- Smith, D. Pallett, C. Pao, P. Price, and A. Rudnicky. In addition, many other people made important contributions to this work and are listed in the Acknowledgements section. The charter of MADCOW was to implement the multi- site data collection paradigm, to monitor the distribution of the data, and to agree on a test paradigm for the multi-site data. The original goals for the data collection activity were to collect 10,000 training utterances and 1,000 test utterances, plus material for a dry-run test to be held in October 1991. Between May 1991 and February 1992, the following data have been collected under the MADCOW effort: 10,400 training utterances collected from 280 speak- ers at 5 sites, with speech and transcriptions dissem- inated to testing sites; • 5,600 utterances annotated with database reference answers, distributed to testing sites; • 300 annotated utterances used for October dry-run test; • 1,000 annotated utterances used for the February 1992 test; • 1,000 utterances set aside for a later test. Significant data collection and evaluation infrastructure were already in place prior to the formation of MAD- COW. This included the definition of the air travel plan- ning task [11], the database (a relational version …

Content truncated to 500 words