Text-Free Prosody-Aware Generative Spoken Language Modeling

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8666 - 8681 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Text-Free Prosody-Aware Generative Spoken Language Modeling Eugene Kharitonov∗, Ann Lee∗, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, Wei-Ning Hsu Facebook AI Research {kharitonov,annl,wnhsu}@fb.com Abstract Speech pre-training has primarily demon- strated efficacy on classification tasks, while its capability of generating novel speech, sim- ilar to how GPT-2 can generate coherent para- graphs, has barely been explored. Generative Spoken Language Modeling (GSLM) (Lakho- tia et al., 2021) is the only prior work ad- dressing the generative aspects of speech pre- training, which replaces text with discov- ered phone-like units for language modeling and shows the ability to generate meaning- ful novel sentences. Unfortunately, despite eliminating the need of text, the units used in GSLM discard most of the prosodic in- formation. Hence, GSLM fails to leverage prosody for better comprehension, and does not generate expressive speech. In this work, we present a prosody-aware generative spo- ken language model (pGSLM). It is composed of a multi-stream transformer language model (MS-TLM) of speech, represented as discov- ered unit and prosodic feature streams, and an adapted HiFi-GAN model converting MS- TLM outputs to waveforms. We devise a se- ries of metrics for prosody modeling and gen- eration, and re-use metrics from GSLM for content modeling. Experimental results show that the pGSLM can utilize prosody to im- prove both prosody and content modeling, and also generate natural, meaningful, and coher- ent speech given a spoken prompt.1 1 Introduction Natural language processing (NLP) has made tremendous progress recently. One of the most sig- nificant findings is that language models (LMs) are natural unsupervised multitask learners (Radford et al., 2018, 2019; Brown et al., 2020) — by simply training a big neural network on next word predic- tion with a large amount of unlabeled text, it learns 1Audio samples can be found at https://speechbot. github.io/pgslm/. Codes and models are avail- able at https://github.com/pytorch/fairseq/ tree/main/examples/textless_nlp/pgslm. to comprehend, answer questions, summarize, and even translate (Radford et al., 2019). Fine-tuning such pre-trained models further leads to the state- of-the-art performance on numerous benchmark tasks (Brown et al., 2020), beating tailor-made mod- els trained from scratch only on labeled data. Given the impressive performance of pre-trained text language models, it is tempting to approach spoken language processing tasks by first transcrib- ing speech into text with an automatic speech recog- nition (ASR) system and then utilizing text-based models for comprehension and generation. How- ever, there are a number of caveats for such a frame- work. First, the majority of the world’s languages are primarily spoken and do not have associated texts in large quantities (Lewis et al., 2016). In practice, this limits the reach of NLP techniques to a fraction of the world’s languages that have a large presence on the web and for which there exists a widely available high quality ASR system. Second, …

Content truncated to 500 words