Optimizing Computer-Assisted Transcription Quality with Iterative User Interfaces

Optimizing Computer-Assisted Transcription Quality with Iterative User Interfaces Matthias Sperber1, Graham Neubig2, Satoshi Nakamura2, Alex Waibel1 1Karlsruhe Institute of Technology, Institute of Anthropomatics and Robotics, Germany 2Nara Institute of Science and Technology, Augmented Human Communications Laboratory, Japan Abstract Computer-assisted transcription promises high-quality speech transcription at reduced costs. This is achieved by limiting human effort to transcribing parts for which automatic transcription quality is insufficient. Our goal is to improve the human transcription quality via appropriate user interface design. We focus on iterative interfaces that allow humans to solve tasks based on an initially given suggestion, in this case an automatic transcription. We conduct a user study that reveals considerable quality gains for three variations of iterative interfaces over a non-iterative from-scratch transcription interface. Our iterative interfaces included post-editing, confidence-enhanced post-editing, and a novel retyping interface. All three yielded similar quality on average, but we found that the proposed retyping interface was less sensitive to the difficulty of the segment, and superior when the automatic transcription of the segment contained relatively many errors. An analysis using mixed-effects models allows us to quantify these and other factors and draw conclusions over which interface design should be chosen in which circumstance. Keywords: Computer-assisted transcription, user interfaces, quality-speed-tradeoff, mixed-effects models 1. Introduction Manual speech transcription by human transcribers has risen in popularity in recent years (Ipeirotis, 2010). This may be surprising, because at the same time automatic speech recognition (ASR) technology has celebrated im- pressive advances. Even so, manual transcription through crowd-sourcing has been established as an attractive alter- native and supplement to fully automatic transcription. In addition, there has also been a rise in computer-assisted transcription, in which automatic transcription is used as a starting point and human intervention is only needed when the ASR produces errors (Rodriguez et al., 2007). Both ap- proaches promise better reliability than ASR, while being more affordable than fully manual expert transcription. Our goal in this paper is to design a computer-assisted transcription user interface such that the outcome quality is optimized while avoiding unnecessary effort. The key inter- face feature we investigate is support for iterative transcrip- tion. This term is borrowed from iterative human computa- tion processes (Little et al., 2010), in which humans solve tasks by improving upon a previously obtained solution. We consider computer-assisted transcription performed in an efficient segment-by-segment fashion, where only low- confidence segments are selected for manual transcription (Roy and Roy, 2009; Sperber et al., 2014b). Our iterative interfaces then provide the initial transcription as created by the ASR as a starting point for each segment, upon which the transcriber improves (cf. Figure 1). The benefit of the iterative interfaces is that the transcriber can simply use the initially correct parts from the ASR as-is, and focus atten- tion on the problematic parts. Ideally, words that were rec- ognized correctly by the ASR will not be changed, reducing the chance of correction errors. In addition, the iterative ap- proach can assist transcription of parts that are difficult for the transcriber …

Content truncated to 500 words