The Hub and Spoke Paradigm for CSR Evaluation

The Hub and Spoke Paradigm for CSR Evaluation Francis Kubala Jerome Bellegarda Jordan Cohen David Pallett Doug Paul Mike Phillips Raja Rajasekaran Fred Richardson Michael Riley Roni Rosenfeld Bob Roth Mitch Weintraub BBN Systems and Technologies IBM T. J. Watson Research Center Institute for Defense Analyses National Institute of Standards and Technology MIT Lincoln Laboratory MIT Laboratory for Computer Science Texas Instruments Boston University AT&T Bell Laboratories Carnegie Mellon University Dragon Systems, Inc. SRI International 1993 Members of the CSR Corpus Coordinating Committee (CCCC) e-mail: cccc@bbn.com ABSTRACT In this paper, we introduce the new paradigm used in the most re- cent ARPA-sponsored Continuous Speech Recognition (CSR) eval- uation and then discuss the important features of the test design. The 1993 CSR evaluation was organized in a novel fashion in an attempt to accomodate research over a broad variety of important problems in CSR while maintaining a clear program-wide research focus. Furthermore, each test component in the evaluation was de- signed as an experiment to extract as much information as possible from the results. The evaluation was centered around a large vocabulary speaker- independent (SI) baseline test, which was required of every partic- ipating site. This test was dubbed the 'Hub' since it was common to all sites and formed the basis for controlled inter-system com- parisons. The Hub test was augmented with a variety of problem-specific optional tests designed to explore a variety of important problems in CSR, mostly involving some kind of mismatch between the training and test conditions. These tests were known as the 'Spokes' since they all could be informatively compared to the Hub, but were otherwise independent. In the first trial of this evaluation paradigm in November, 1993, 11 research groups participated, yielding a rich array of comparative and contrastive results, all calibrated to the current state of the art in large vocabulary CSR. 1. Introduction Since 1986, ARPA has sponsored periodic formal evaluations of CSR technology. From the beginning, these evaluations were dis- tinguished by a well-defined test that was required of all partici- pants within a specified window of time. Most importantly, the test definition remained stable over several years so that sustained effort could be made toward improved performance and so that any improvement made over time could be demonstrated convincingly. These were important features of a series of evaluations based on the well-known Resource Management (RM) corpus. Those evalu- ations were highly regarded for the competitive stimulus they pro- duced, resulting in the rapid assimilation of new techniques across the CSR community worldwide. When the ARPA CSR community began designing a testbed cor- pus for large-vocabulary recognition to replace RM, one of the shortcomings addressed was its lack of support for the variety of important research interests that existed within the community at the time. Active research was already underway in adaptation to speaker, domain, dialect, and in compensation for mismatch in mi- crophone, environment, and speaking style, but none of this was supported by the RM corpus. The ARPA-chartered CSR Corpus …

Content truncated to 500 words