Simulating ASR errors for training SLU systems

Simulating ASR errors for training SLU systems Edwin Simonnet, Sahar Ghannay, Nathalie Camelin, Yannick Est`eve LIUM, Le Mans University, France firstname.lastname@univ-lemans.fr Abstract This paper presents an approach to simulate automatic speech recognition (ASR) errors from manual transcriptions and describes how it can be used to improve the performance of spoken language understanding (SLU) systems. In particular, we point out that this noising process is very usefull to obtain a more robust SLU system to ASR errors in case of insufficient training data or more if ASR transcriptions are not available during the training of the SLU model. The proposed method is based on the use of both acoustic and linguistic word embeddings in order to define a similarity measure between words dedicated to predict ASR confusions. Actually, we assume that words acoustically and linguistically close are the ones confused by an ASR system. By using this similarity measure in order to randomly substitute correct words by potentially confusing words in manual annotations used to train CRF- or neural- based SLU systems, we augment the training corpus with these new noisy data. Experiments were carried on the French MEDIA corpus focusing on hotel reservation. They show that this approach significantly improves SLU system performance with a relative reduction of 21.2% of concept/value error rate (CVER), particularly when the SLU system is based on a neural approach (re- duction of 22.4% of CVER). A comparison to a naive noising approach shows that the proposed noising approach is particularly relevant. Keywords: spoken language understanding, data augmentation, noising, automatic speech recognition, errors 1. Introduction Spoken language understanding (SLU) consists in extract- ing semantic information from speech, and can refer to dif- ferent tasks. In (De Mori, 2007), the author defines it as ”(...) the interpretation of signs conveyed by a speech sig- nal”. Similar to previous works from other authors (Hahn et al., 2011; Mesnil et al., 2015), the SLU task targeted in this paper consists in automatically extracting semantic concepts and concept/values pairs from the automatic tran- scriptions in order to feed a dialogue manager. This task can also be perceived as a slot filling task. Usually, SLU needs first an automatic transcription of user utterances thanks to an automatic speech recognition (ASR) system. These recognized words are then analyzed in or- der to extract their meanings. Even best SLU systems see their performance drop when making the transition from processing manual transcriptions to automatic ones, since ASR errors make the SLU task harder. In order to reduce this unavoidable performance decline, it would be relevant to prepare SLU systems to ASR errors during their train- ing. For instance, it is known that Spoken Dialog Sys- tem (SDS) applied to automatic transcriptions perform bet- ter when they are trained on automatic transcriptions rather than manual ones. Since large automatic transcription cor- pora needed for SDS training are rare, some approaches have been presented in order to simulate ASR errors to train these SDS (Pietquin and Beaufort, 2005; Schatzmann et al., 2007). Such ASR …

Content truncated to 500 words