Generative Spoken Dialogue Language Modeling

Generative Spoken Dialogue Language Modeling Tu Anh Nguyen⋄,†, Eugene Kharitonov⋄∗, Jade Copet⋄, Yossi Adi§, Wei-Ning Hsu¶, Ali Elkahky¶, Paden Tomasello¶, Robin Algayres⋄∗, Benoˆıt Sagot†, Abdelrahman Mohamed⋄∗, Emmanuel Dupoux⋄,‡ ⋄Meta AI Research, France †Inria, Paris, France §Meta AI Research, Isra¨el ¶Meta AI Research, United States ‡EHESS, ENS-PSL, CNRS, Paris, France {ntuanh, abdo, dpx}@meta.com Abstract We introduce dGSLM, the first ‘‘textless’’ model able to generate audio samples of nat- uralistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery cou- pled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter, and other paralinguistic signals in the two channels simultaneously and repro- duces more naturalistic and fluid turn taking compared to a text-based cascaded model.1,2 1 Introduction In natural conversations, speakers spontaneously coordinate who is currently speaking and when the other person will speak next. As a result, conver- sations end up being a fluent succession of turns without much overlapping speech or long stretches of silence. Of course, silences and overlaps also occur naturally and they carry significant informa- tion which is interpreted within the conversation setting. For instance, when overlapping speech occurs it often contains content-neutral verbal in- formation (e.g., ‘‘hmm’’, ‘‘yeah’’) or non-verbal vocalization (e.g., laughter), used to convey a lis- tening attitude (back-chanelling) (Yngve, 1970; Schegloff, 1982). Short silences between turns do occur and show both cross-cultural variations and universal dependence on dialogue related vari- ables, for instance, straight and positive answers to questions are typically faster than non-responses or negative responses (Stivers et al., 2009). ∗Work done while at Meta. 1Generation samples can be found at https:// speechbot.github.io/dgslm. at2Code and pre-trained models will be made available https://github.com/facebookresearch /fairseq/tree/main/examples/textless_nlp /dgslm. All of this turn-taking coordination is natural to humans, and starts to be learned at an early age by infants (Nguyen et al., 2022). In con- trast, it remains a challenging area of research in human/machine interactions (Skantze, 2021). One of the reason is that much of the research into natural dialogue modeling is taking place with text-based interfaces. Here, the coordination problem is primarily focused on semantic coher- ence and appropriateness of the artificial agent in interaction with a human (see Ni et al., 2021, for a review). The turn-taking problem itself is being taken care of by an artificially imposed walkie talkie arrangement; each agent is writing in turn and signalling the end of its turn by pressing carriage return. Within speech-based systems, it is very similar, as current spoken assistants like Siri or Alexa are triggered by a predetermined wake word, and wait for the end of an utterance followed by sufficient silence to segment the turns of the human inter- locutor. This may give rise to slow and unnatural conversations. In fact, in human-human conver- sation, pauses within speaker turns tend to be on average longer than gaps between speaker turns (Brady, 1968; Ten Bosch et al., 2005; …

Content truncated to 500 words