Learning Robust and Multilingual Speech Representations

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1182–1192 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1182 Learning Robust and Multilingual Speech Representations Kazuya Kawakami♣♠ Luyu Wang♣ Chris Dyer♣ Phil Blunsom♣♠ Aaron van den Oord♣ ♣DeepMind, London, UK ♠Department of Computer Science, University of Oxford, Oxford, UK {kawakamik,luyuwang,cdyer,pblunsom,avdnoord}@google.com Abstract Unsupervised speech representation learning has shown remarkable success at finding rep- resentations that correlate with phonetic struc- tures and improve downstream speech recog- nition performance. However, most research has been focused on evaluating the representa- tions in terms of their ability to improve the performance of speech recognition systems on read English (e.g. Wall Street Journal and Lib- riSpeech). This evaluation methodology over- looks two important desiderata that speech rep- resentations should have: robustness to do- main shifts and transferability to other lan- guages. In this paper we learn representa- tions from up to 8000 hours of diverse and noisy speech data and evaluate the represen- tations by looking at their robustness to do- main shifts and their ability to improve recog- nition performance in many languages. We find that our representations confer significant robustness advantages to the resulting recogni- tion systems: we see significant improvements in out-of-domain transfer relative to baseline feature sets and the features likewise provide improvements in 25 phonetically diverse lan- guages. 1 Introduction The input representation of machine learning model strongly determines the difficulty faced by the learning algorithm, how much data the learner will require to find a good solution, and whether the learner generalizes out of sample and out of the domain of the training data. Representations (or features) that encode relevant information about data enable models to achieve good performance on downstream tasks, while representations that are invariant to factors that are not relevant to down- stream tasks can further improve generalization. Traditionally, many invariances were hard-coded in feature extraction methods. For example, in image representations, geometric and photometric invari- ance has been investigated (Mundy et al., 1992; Van De Weijer et al., 2005). For acoustic represen- tations, standard MFCC features are sensitive to additive noise and many modifications have been proposed to overcome those limitations (Dev and Bansal, 2010; Kumar et al., 2011). Recently, unsupervised representation learning algorithms have shown significant improvements at learning representations that correlate well with phonetic structure (van den Oord et al., 2018; Kahn et al., 2019b) and improving downstream speech recognition performance (Schneider et al., 2019; Baevski et al., 2019). Most of this work focused on learning representations from read English speech (from the LibriSpeech and LibriVox datasets) and evaluating the features when used to recognize speech in a rather similar domain (read English text). However, this approach to evaluation fails to test for the invariances that we would like good speech representations to have: robustness to do- main shifts and transferability to other languages. In our experiments we learn representations from 8000 hours of diverse and noisy speech, using an extended version of contrastive predictive coding model: bidirectional predictive models with …

Content truncated to 500 words