Analyzing Learned Representations of a Deep ASR Performance Prediction Model

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 9–15 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 9 Analyzing Learned Representations of a Deep ASR Performance Prediction Model Zied Elloumi1,2 Laurent Besacier2 Olivier Galibert1 1Laboratoire national de m´etrologie et d’essais (LNE) , France 2Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, F-38000 Grenoble, France firstname.name@lne.fr firstname.name@univ-grenoble-alpes.fr Benjamin Lecouteux2 Abstract This paper addresses a relatively new task: prediction of ASR performance on unseen broadcast programs. In a previous paper, we presented an ASR performance prediction sys- tem using CNNs that encode both text (ASR transcript) and speech, in order to predict word error rate. This work is dedicated to the anal- ysis of speech signal embeddings and text em- beddings learnt by the CNN while training our prediction model. We try to better understand which information is captured by the deep model and its relation with different condition- ing factors. It is shown that hidden layers con- vey a clear signal about speech style, accent and broadcast type. We then try to leverage these 3 types of information at training time through multi-task learning. Our experiments show that this allows to train slightly more ef- ficient ASR performance prediction systems that - in addition - simultaneously tag the an- alyzed utterances according to their speech style, accent and broadcast program origin. 1 Introduction Predicting automatic speech recognition (ASR) performance on unseen speech recordings is an important Grail of speech research. In a previ- ous paper (Elloumi et al., 2018), we presented a framework for modeling and evaluating ASR performance prediction on unseen broadcast pro- grams. CNNs were very efficient encoding both text (ASR transcript) and speech to predict ASR word error rate (WER). However, while achiev- ing state-of-the-art performance prediction results, our CNN approach is more difficult to understand compared to conventional approaches based on engineered features such as TransRater1 for in- stance. This lack of interpretability of the repre- sentations learned by deep neural networks is a 1https://github.com/hlt-mt/TranscRater general problem in AI. Recent papers started to address this issue and analyzed hidden represen- tations learned during training of different natu- ral language processing models (Mohamed et al., 2012; Wu and King, 2016; Belinkov and Glass, 2017; Shi et al., 2016; Belinkov et al., 2017; Wang et al., 2017). Contribution. This work is dedicated to the analysis of speech signal embeddings and text em- beddings learnt by the CNN during training of our ASR performance prediction model. Our goal is to better understand which information is cap- tured by the deep model and its relation with con- ditioning factors such as speech style, accent or broadcast program type. For this, we use a data set presented in (Elloumi et al., 2018) which con- tains a large amount of speech utterances taken from various collections of French broadcast pro- grams. Following a methodology similar to (Be- linkov and Glass, 2017), our deep performance prediction model is used to generate utterance level features that are …

Content truncated to 500 words