Text Alignment for Real-Time Crowd Captioning

Proceedings of NAACL-HLT 2013, pages 201–210, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Text Alignment for Real-Time Crowd Captioning Iftekhar Naim, Daniel Gildea, Walter Lasecki and Jeffrey P. Bigham Department of Computer Science University of Rochester Rochester, NY 14627 Abstract The primary way of providing real-time cap- tioning for deaf and hard of hearing people is to employ expensive professional stenogra- phers who can type as fast as natural speak- ing rates. Recent work has shown that a feasible alternative is to combine the partial captions of ordinary typists, each of whom types part of what they hear. In this paper, we describe an improved method for combin- ing partial captions into a final output based on weighted A∗ search and multiple sequence alignment (MSA). In contrast to prior work, our method allows the tradeoff between accu- racy and speed to be tuned, and provides for- mal error bounds. Our method outperforms the current state-of-the-art on Word Error Rate (WER) (29.6%), BLEU Score (41.4%), and F-measure (36.9%). The end goal is for these captions to be used by people, and so we also compare how these metrics correlate with the judgments of 50 study participants, which may assist others looking to make fur- ther progress on this problem. 1 Introduction Real-time captioning provides deaf or hard of hear- ing people access to speech in mainstream class- rooms, at public events, and on live television. To maintain consistency between the captions being read and other visual cues, the latency between when a word was said and when it is displayed must be under five seconds. The most common approach to real-time captioning is to recruit a trained stenogra- pher with a special purpose phonetic keyboard, who transcribes the speech to text within approximately 5 seconds. Unfortunately, professional captionists are quite expensive ($150 per hour), must be recruited in blocks of an hour or more, and are difficult to sched- ule on short notice. Automatic speech recognition (ASR) (Saraclar et al., 2002) attempts to solve this ������������������������������ �������� ��������������������������� ��������������������������� �������������������������������������������� ������������� ��������������������������� ��������������������������� �� ������������������������������ �� ��������������������������� �� Figure 1: General layout of crowd captioning systems. Captionists (C1, C2, C3) submit partial captions that are automatically combined into a high-quality output. problem by converting speech to text completely au- tomatically. However, the accuracy of ASR quickly plummets to below 30% when used on an untrained speaker’s voice, in a new environment, or in the ab- sence of a high quality microphone (Wald, 2006b). An alternative approach is to combine the efforts of multiple non-expert captionists (anyone who can type) (Lasecki et al., 2012; Lasecki and Bigham, 2012; Lasecki et al., 2013). In this approach, mul- tiple non-expert human workers transcribe an audio stream containing speech in real-time, and their par- tial input is combined to produce a final transcript (see Figure 1). This approach has been shown to dramatically outperform ASR in terms of both accu- racy and Word Error Rate (WER), even when us- ing captionists drawn …

Content truncated to 500 words