Improved Speech Representations with Multi-Target Autoregressive Predictive Coding

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2353–2358 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2353 Improved Speech Representations with Multi-Target Autoregressive Predictive Coding Yu-An Chung, James Glass Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, MA 02139, USA {andyyuan,glass}@mit.edu Abstract Training objectives based on predictive coding have recently been shown to be very effective at learning meaningful representations from unlabeled speech. One example is Autoregres- sive Predictive Coding (Chung et al., 2019), which trains an autoregressive RNN to gen- erate an unseen future frame given a context such as recent past frames. The basic hypoth- esis of these approaches is that hidden states that can accurately predict future frames are a useful representation for many downstream tasks. In this paper we extend this hypothe- sis and aim to enrich the information encoded in the hidden states by training the model to make more accurate future predictions. We propose an auxiliary objective that serves as a regularization to improve generalization of the future frame prediction task. Experimental re- sults on phonetic classification, speech recog- nition, and speech translation not only support the hypothesis, but also demonstrate the effec- tiveness of our approach in learning represen- tations that contain richer phonetic content. 1 Introduction Unsupervised speech representation learning, which aims to learn a function that transforms sur- face features, such as audio waveforms or spectro- grams, to higher-level representations using only unlabeled speech, has received great attention re- cently (Baevski et al., 2019, 2020; Liu et al., 2020; Song et al., 2019; Jiang et al., 2019; Schneider et al., 2019; Chorowski et al., 2019; Pascual et al., 2019; Oord et al., 2018; Kamper, 2019; Chen et al., 2018; Chung and Glass, 2018; Chung et al., 2018; Milde and Biemann, 2018; Chung et al., 2016; Hsu et al., 2017). A large portion of these approaches leverage self-supervised training, where the learn- ing target is generated from the input itself, and thus can train a model in a supervised manner. Chung et al. (2019) propose a method called Au- toregressive Predictive Coding (APC), which trains an RNN to predict a future frame that is n steps ahead of the current position given a context such as the past frames. The training target can be eas- ily generated by right-shifting the input by n steps. Their intuition is that the model is required to pro- duce a good summarization of the past and encode such knowledge in the hidden states so as to accom- plish the objective. After training, the RNN hidden states are taken as the learned representations, and are shown to contain speech information such as phonetic and speaker content that are useful in a variety of speech tasks (Chung and Glass, 2020). Following their intuition, in this work we aim to improve the generalization of the future frame pre- diction task by adding an auxiliary objective that serves as a regularization. We empirically demon- strate the effectiveness of our …

Content truncated to 500 words