Collection and Analysis of Data From Real Users: Implications for Speech Recognition/Understanding Systems

COLLECTION AND ANALYSIS OF DATA FROM REAL USERS: IMPLICATIONS FOR SPEECH RECOGNITION/UNDERSTANDING SYSTEMS Judith Spitz and the Artificial Intelligence Speech Technology Group NYNEX Science and Technology, Research and Development 500 Westchester Avenue White Plains, New York 10604 ABSTRACT Performance estimates given for speech recognition/understanding systems are typically based on the assumption that users will behave in ways similar to the observed behavior of laboratory volunteers. This includes the acoustic/phonetic characteristics of the spcech they produce as well as their willingness and ability to constrain their input to the device according to instructions. Since speech recognition devices often do not perform as well in the field as they do in the laboratory, analyses of real user behavior have been undertaken. The results of several field trials suggest that real user compliance with instructions is dramatically affected by the particular details of the prompts supplied to the user. A significant amount of real user speech data has been collected during these trials (34,000 uuerances, 29 hours of data). These speech databases are described along with the results of an experiment comparing the performance of a speech recognition system on real user vs. laboratory speech. INTRODUCTION Speech recognition/understanding systems will ultimately establish their usefulness by working well under real application conditions. Success in the field will depend not only on the technology itself but also on the behavior of real users. Real user behavior can be characterized in terms of 1. what people say and 2. how they say it. What people say: Real user compliance Until the advent of a high performance continuous speech, unconstrained vocabulary/grammar, interactive speech understanding system, users must constrain their spoken interactions with speech recognition/understanding systems. Constraints may require speaking words in isolation, conforming to a limited vocabulary or grammar, restricting queries to a particular knowledge domain, etc. The users' willingness to comply with instructions specifying these constraints will determine the success of the technology. If users are willing or even able to confine themselves to one of two words (e.g., yes or no), a two-word speech recognition system may succeed. If users are non-compliant (e.g., say the target words embedded in phrases, say synonyms of the target words, reject the service as a result of the constraining instructions), the technology will fail in the field; despite high accuracy laboratory performance. How compliant are real users? The answer may be application- specific, dependent on particulars such as 1. frequency of repeat usage of the system, 2. motivation of the users, 3. cost of an error, 4. nature of the constraint, etc. It would be useful to understand the factors that predict compliance, and to know whether generalizations can be made across applications. In addition, it would be useful to have a better understanding of how to maximize user compliance. Moreover, there is value in analyzing non-compliant behavior. To the extent that non-compliance takes the form of choosing synonyms of the target words, the recognizer's vocabulary must be expanded. If non-compliance takes the form of embedding the target word …

Content truncated to 500 words