Multi-Site Data Collection and Evaluation in Spoken Language Understanding

Multi-Site Data Collection and Evaluation in Spoken Language Understanding L. Hirschman, M. Bates, D. Dahl, W. Fisher, ]. Garofolo, D. Pallett, K. Hunicke-Smith, P. Price, A. Rudnicky, and E. Tzoukermann* Contact: Lynette Hirschman NE43-643 Spoken Language Systems Group MIT Laboratory for Computer Science, Cambridge, MA 02139 e-mail: lynet te@goldilocks.lcs.mit.edu ABSTRACT The Air Travel Information System (ATIS) domain serves as the common task for DARPA spoken language system re- search and development. The approaches and results possi- ble in this rapidly growing area are structured by available corpora, annotations of that data, and evaluation methods. Coordination of this crucial infrastructure is the charter of the Multi-Site ATIS Data COllection Working group (MAD- COW). We focus here on selection of training and test data, evaluation of language understanding, and the continuing search for evaluation methods that will correlate well with expected performance of the technology in applications. 1. INTRODUCTION Data availability and evaluation procedures structure re- search possibilities: the type and amount of training data affects the performance of existing algorithms and limits the development of new algorithms; and evalua- tion procedures document progress, and force research choices in a world of limited resources. The recent rapid progress in spoken language understanding owes much to our success in collecting and distributing a large corpus of speech, transcriptions, and associated materials based on human-machine interactions in the air travel domain. A tight feedback loop between evaluation methodology and evaluation results has encouraged incremental ex- tension to the evaluation methodology, to keep pace with the technology development. The paper reports on the data collection and evaluation efforts co-ordinated by MADCOW over the past year. The multi-site data collection paradigm [3, 4] distributes the burden of data collection, provides data rapidly, ed- ucates multiple sites about data collection issues, and results in a more diverse pool of data than could be ob- tained with a single collection site. The resulting data represents a wide range of variability in speaker char- acteristics, speech style, language style and interaction style. It has allowed individual sites to experiment with data collection methods: by replacing various system *This paper was written the auspices of the Multi-Site ATIS Data Collection Working group (MADCOW). In addition to the authors, many other people, listed under the Acknowledgements section, made important contributions to this work. components with a human, we collect the kind of data we can aim for in the future, while completely automated systems help us to focus on the major current issues in system accuracy and speed. Sites have also exper- imented with interface strategies: spoken output only, tabular output only, response summaries, spoken and written paraphrase, and system initiative may be more or less appropriate for different users and different tasks and all can dramatically affect the resulting data. MADCOW's recent accomplishments include: • Release of 14,000 utterances for training and test, including speech and transcriptions; • Release of annotations for almost I0,000 of these utterances (7500 training utterances and three test sets of 2300 utterances total), balanced by …

Content truncated to 500 words