MOCCA: Measure of Confidence for Corpus Analysis - Automatic Reliability Check of Transcript and Automatic Segmentation

MOCCA: Measure of Confidence for Corpus Analysis - Automatic Reliability Check of Transcript and Automatic Segmentation Thomas Kisler, Florian Schiel Institute of Phonetics and Speech Processing Schellingstr. 3, 80799 M¨unchen, Germany {kisler, schiel}@bas.uni-muenchen.de Abstract The production of speech corpora typically involves manual labor to verify and correct the output of automatic transcription/segmentation processes. This study investigates the possibility of speeding up this correction process using techniques borrowed from automatic speech recognition to predict the location of transcription or segmentation errors in the signal. This was achieved with functionals of features derived from a typical Hidden Markov Model (HMM)-based speech segmentation system and a classification/regression approach based on Support Vector Machine (SVM)/Support Vector Regression (SVR) and Random Forest (RF). Classifiers were tuned in a 10-fold cross validation on an annotated corpus of spontaneous speech. Tests on an independent speech corpus from a different domain showed that transcription errors were predicted with an accuracy of 78% using an SVM, while segmentation errors were predicted in the form of an overlap-measure which showed a Pearson correlation of 0.64 to a ground truth using SVR. The methods described here will be implemented as free-to-use Common Language and Resources and Technology Infrastucture (CLARIN) web services. Keywords: automatic segmentation, MAUS, confidence measure 1. Introduction The creation of a new speech corpus typically involves three major steps: (1) recording, (2) (orthographic) tran- scription and (3) alignment of the transcription to the recorded signal, referred to hereafter as segmentation and labeling (S&L). The quality of these three production steps more or less defines the usefulness of the speech resource. The transcription of a speech recording (2) can be done ei- ther manually or via Automatic Speech Recognition (ASR). In both cases the transcription may contain errors in the form of deviations between the transcribed words and the words that were actually spoken. The S&L (3) can also be done either manually or automatically based on the tran- script created at step (2). Since manual S&L is more time- consuming than (2) (slower by a factor of around 20 to 100), step (3) is often first done automatically (applying text-to-speech alignment or similar techniques) and then manually corrected afterwards. Both tasks – the manual correction of the transcript or the manual correction of the S&L – are expensive and time-consuming because every part of every utterance must be checked manually. This study is concerned with the automatic and reliable de- tection of errors in the S&L either caused by falsely tran- scribed words1 or by errors of the applied S&L system: Measure of Confidence for Corpus Analysis (MOCCA). More specifically, MOCCA consists of two methods for the reliable detection of word errors in the transcription and for the word-by-word estimation of the quality of the S&L in the form of an overlap measure. Both MOCCA meth- ods can be used to automatically identify parts in the S&L where ’something went wrong’, and thus facilitate the man- ual correction task. The estimation of the correctness of a word label in …

Content truncated to 500 words