Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 465–477 July 13-14, 2023 c⃝2023 Association for Computational Linguistics Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling Itai Gat♢, Felix Kreuk♢, Tu Anh Nguyen♢, Ann Lee♢, Jade Copet♢, Gabriel Synnaeve♢, Emmanuel Dupoux♠,♢, Yossi Adi♡,♢ ♢FAIR Team, Meta AI Research ♠ENS, INRIA, INSERM, UPEC, PSL Research University ♡The Hebrew University of Jerusalem Abstract Generative Spoken Language Modeling re- search focuses on optimizing speech Language Models (LMs) using raw audio recordings with- out accessing any textual supervision. Such speech LMs usually operate over discrete units obtained from quantizing internal representa- tions of self-supervised models. Although such units show impressive modeling results, their robustness capabilities have not been exten- sively investigated. This work focuses on im- proving the invariance of discrete input rep- resentations to non-spoken augmentations for generative spoken language modeling. First, we formally define how to measure the robust- ness of such representations to various signal variations that do not alter the spoken infor- mation (e.g., time-stretch). Next, we empiri- cally demonstrate how current state-of-the-art representation models lack robustness to such variations. To overcome this, we propose an effective and efficient method to learn invari- ant discrete speech representation for genera- tive spoken language modeling. The proposed approach is based on applying a set of signal transformations to the speech signal and op- timizing the model using an iterative pseudo- labeling scheme. Our method significantly im- proves over the evaluated baselines when con- sidering encoding and modeling metrics. We additionally evaluate our method on the speech- to-speech translation task, considering Spanish- English and French-English translations, and show the proposed approach outperforms the evaluated baselines. 1 Introduction Self-supervised speech models were shown to learn effective representations for various downstream tasks (Hsu et al., 2021; Chen et al., 2022; Baevski et al., 2020). These models were mainly evaluated on discriminative tasks, such as automatic speech recognition, speaker verification, intent classifica- tion, etc. (Yang et al., 2021). Recently, Lakhotia et al. (2021) demonstrated that such self-supervised learning (SSL) representations can be used for Gen- erative Spoken Language Modeling. Generative Spoken Language Modeling (GSLM) is the task of learning the acoustic and linguistic characteristics of a language from raw audio. In other words, a discrete representation of the au- dio signal is being learned. A common practice is to extract continuous representation using an SSL model, then apply vector quantization, usu- ally using the k-means algorithm (Lakhotia et al., 2021; Kharitonov et al., 2021a; Borsos et al., 2022). Then a speech-language model is trained on top of the obtained representation. Finally, a neural vocoder converts the output units to raw audio. As the discrete speech representation often operates over units extracted over relatively short windows (e.g., 20ms), sequences can be long and contain repetitions, e.g., 10 11 11 11 21 32 32 32 21. Preliminary studies have found that removing se- quential repetitions of units improves performance, hence applying it universally (Lakhotia et al., 2021). For example, a pseudo-text 10 11 …

Content truncated to 500 words