Cheap, Fast and Good Enough: Automatic Speech Recognition with Non-Expert Transcription

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 207–215, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Cheap, Fast and Good Enough: Automatic Speech Recognition with Non-Expert Transcription Scott Novotney and Chris Callison-Burch Center for Language and Speech Processing Johns Hopkins University snovotne@bbn.com ccb@jhu.edu Abstract Deploying an automatic speech recogni- tion system with reasonable performance requires expensive and time-consuming in-domain transcription. Previous work demonstrated that non-professional anno- tation through Amazon’s Mechanical Turk can match professional quality. We use Mechanical Turk to transcribe conversa- tional speech for as little as one thir- tieth the cost of professional transcrip- tion. The higher disagreement of non- professional transcribers does not have a significant effect on system performance. While previous work demonstrated that redundant transcription can improve data quality, we found that resources are bet- ter spent collecting more data. Finally, we describe a quality control method without needing professional transcription. 1 Introduction Successful speech recognition depends on huge investments in data collection. Even after train- ing on 2000+ hours of transcribed conversa- tional speech, over a billion words of language modeling text, and hand-crafted pronunciation dictionaries, state of the art systems still have an error rate of around 15% for English (Prasad et al., 2005) Transcribing the large volumes of data required for Large Vocabulary Continuous Speech Recognition (LVCSR) of new languages appears prohibitively expensive. Recent work has shown that Amazon’s Mechanical Turk1 can 1http://www.mturk.com be used to cheaply create data for other nat- ural language processing applications (Snow et al., 2008; Zaidan and Callison-Burch, 2009; Mc- Graw et al., 2009). In this paper we focus on reducing the cost of transcribing conversa- tional telephone speech (CTS) data. Previous measurements of Mechanical Turk stopped at agreement/disagreement with professional an- notation. We take the next logical step and measure performance on systems trained with non-professional transcription. Mechanical Turk is an online labor mar- ket where workers (or Turkers) perform simple tasks called Human Intelligence Tasks (HITs) for small amounts of money – frequently as lit- tle as $0.01 per HIT. Since HITs can be tasks that are difficult for computers, but easy for hu- mans, they are ideal for natural language pro- cessing tasks (Snow et al., 2008). Mechanical Turk has even spawned a business that special- izes in manual speech transcription.2 Automatic speech recognition (ASR) of con- versational speech is an extremely difficult prob- lem. Characteristics like rapid speech, pho- netic reductions and speaking style limit the value of non-CTS data, necessitating in-domain transcription. Even a few hours of transcrip- tion is sufficient to bootstrap with unsupervised methods like self-training (Lamel et al., 2002). The speech community has built effective down- stream solutions for the past twenty years de- spite imperfect recognition. In topic classifi- cation, 90% accuracy is possible on conversa- tional data even with 80%+ word error rate 2http://castingwords.com/ 207 (WER) (Gillick et al., 1993). Other successful tasks include information retrieval from speech (Miller et al., 2007) and spoken dialogue pro- …

Content truncated to 500 words