The Design for the Wall Street Journal-based CSR Corpus

The Design for the Wall Street Journal-based CSR Corpus* Douglas B. Paul Lincoln Laboratory, MIT Lexington, Ma. 02173 and Janet M. Baker Dragon Systems, Inc. 320 Nevada St. Newton, Ma. 02160 ABSTRACT The DARPA Spoken Language System (SLS) community has long taken a leadership position in designing, imple- menting, and globally distributing significant speech corpora widely used for advancing speech recognition research. The Wall Street Journal (WSJ) CSR Corpus described here is the newest addition to this valuable set of resources. In contrast to previous corpora, the WSJ corpus will provide DARPA its first general-purpose English, large vocabulary, natural language, high perplexity, corpus containing signif- icant quantities of both speech data (400 hrs.) and text data (47M words), thereby providing a means to integrate speech recognition and natural language processing in ap- plication domains with high potential practical value. This paper presents the motivating goals, acoustic data design, text processing steps, lexicons, and testing paradigms incor- porated into the multi-faceted WSJ CSR Corpus. INTRODUCTION As spoken language technology progresses and goals expand, progressively larger, and more challenging corpora need to be created to support advanced research. The SLS DARPA 1994 goals are ambitious, focusing on cooperative speak- ers, generating goal-directed, spontaneous continuous speech, in speaker-adaptive and speaker-independent modes, for ex- pandable vocabularies (5000 or more words active), moder- ate perplexity (100-200), with integrated speech and natural language processing, for speakers in a moderate noise envi- ronment, using multiple types of microphones, engaged in command/database and dictation applications. In contrast to typical command/database applications, dictation (i.e. in- teractive speech-driven word processing) tasks focus on coop- erative speakers (e.g. speaker dependent/adaptlve sustained usage) who generate continuous speech (usually in a some- what careful fashion to facilitate accurate transcription) ver- balizing their words and sentence punctuation. The existing Resource Management[15] and subsequent Air Travel Infor- mation System[16] corpora target specific database inquiry tasks, characterized by medium vocabularies (<1500 words) with language model perplexities ranging from 9 to 60. The WSJ corpus described here is designed to advance CSR tech- nology and support the 1994 SLS research goals. A similar read speech corpus in the French language has been success- *This work was sponsored by the Defense Advanced Research Projects Agency. The views expressed are those of the authors and do not reflect the official policy or position of the U.S. Government. fully completed using text from the newspaper Le Monde[5]. Commencing with serious contractor concerns regarding suit- able CSR corpora[12] starting in the mid 1980's, the DARPA SLS Coordinating Committee started considering new cor- pora requirements in early 1990, with the subsequent for- mation of the CSR Corpus Committee, culminating in the WSJ Corpus design. The CSR Corpus Committee mem- bers include J.M. Baker (Dragon, chair), F. Kubala (BBN), D. Pallett (NIST), D. Paul (LL), M. Phillips (MIT), M. Picheny (IBM), R. Rajasekran (TI), B. Weide (CMU), M. Weintraub (SRI), and 3. Wilpon (ATT). A survey taken of the DARPA contractors for CSR research interests dis- closed highly diverse, often opposing views of …

Content truncated to 500 words