Introspection for convolutional automatic speech recognition

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 187–199 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 187 Introspection for convolutional automatic speech recognition Andreas Krug and Sebastian Stober University of Potsdam, Research Focus Cognitive Sciences Karl-Liebknecht-Str. 24/25, 14476 Potsdam, Germany {ankrug, sstober}@uni-potsdam.de Abstract Artificial Neural Networks (ANNs) have expe- rienced great success in the past few years. The increasing complexity of these models leads to less understanding about their decision processes. Therefore, introspection techniques have been proposed, mostly for images as input data. Patterns or relevant regions in images can be intuitively interpreted by a human observer. This is not the case for more complex data like speech recordings. In this work, we investigate the appli- cation of common introspection techniques from computer vision to an Automatic Speech Recog- nition (ASR) task. To this end, we use a model similar to image classification, which predicts let- ters from spectrograms. We show difficulties in applying image introspection to ASR. To tackle these problems, we propose normalized aver- aging of aligned inputs (NAvAI): a data-driven method to reveal learned patterns for prediction of specific classes. Our method integrates information from many data examples through local introspection techniques for Convolutional Neural Networks (CNNs). We demonstrate that our method provides better interpretability of letter-specific patterns than existing methods. 1 Introduction Artificial Neural Networks (ANNs) perform incredibly well in many fields of application, even out- performing humans. In particular, deep learning (DL) has been used with great success in a variety of tasks. The most successful applications of DL are in computer vision, like image classification (Krizhevsky et al., 2012) or segmentation (Chen et al., 2014). Moreover, DL performs well in audio processing, like automatic speech recognition (Bahdanau et al., 2016) or machine translation (Wu et al., 2016). One reason for the success of these models is the increase in their complexity by implementing deeper or wider network layers (Szegedy et al., 2015). While this allows the model to learn more complex patterns for solving its task, it is becoming more difficult to interpret how it accomplishes it (Yosinski et al., 2015). Several introspection techniques were proposed to shed light on the decision processes in ANNs (Zeiler and Fergus 2014, Springenberg et al. 2014, Selvaraju et al. 2016). However, most of them come with restrictions on the network architecture or the type of task that is solved. In particular, most methods focus on interpretability of ANNs in computer vision. The reason for this is that evaluating the results from introspection techniques on images is intuitive for a person. This is not the case for more complex data like audio waveforms or multi- channel data like Electroencephalography (EEG) recordings. Applying introspection techniques from computer vision to this kind of data is possible, but evaluating the results is hard, as a human expert cannot easily interpret the input data in the first place. In this work, we investigate the application of several introspection techniques to the domain …

Content truncated to 500 words