LibriS2S: A German-English Speech-to-Speech Translation Corpus

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 928–935 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 928 LibriS2S: A German-English Speech-to-Speech Translation Corpus Pedro Jeuris and Jan Niehues Department of Data Science and Knowledge Engineering, Maastricht University Maastricht, The Netherlands p.jeuris@student.maastrichtuniversity.nl, jan.niehues@maastrichtuniversity.nl, Abstract Recently, we have seen an increasing interest in the area of speech-to-text translation. This has led to astonishing improvements in this area. In contrast, the activities in the area of speech-to-speech translation is still limited, although it is essential to overcome the language barrier. We believe that one of the limiting factors is the availability of appropriate training data. We address this issue by creating LibriS2S, to our knowledge the first publicly available speech-to-speech training corpus between German and English. For this corpus, we used independently created audio for German and English leading to an unbiased pronunciation of the text in both languages. This allows the creation of a new text-to-speech and speech-to-speech translation model that directly learns to generate the speech signal based on the pronunciation of the source language. Using this created corpus, we propose Text-to-Speech models based on the example of the recently proposed FastSpeech 2 model that integrates source language information. We do this by adapting the model to take information such as the pitch, energy or transcript from the source speech as additional input. Keywords: Speech-to-Speech translation, Speech synthesis, Dataset creation 1. Introduction As the world becomes more and more connected, both online and offline, the need for language translation systems rises to overcome the large existing language barrier (Hutchins, 2003; Nakamura, 2009). In the past few years the advances made within the Machine Translation (MT) domain have been significant. Com- puters have become better at translating text between two languages and can even translate speech to text in another language by using a single model (Bahdanau et al., 2016). To solve the Speech-to-Speech Transla- tion (S2ST) problem, in which a spoken phrase needs to be translated and spoken aloud in a target language, the problem is typically broken into three steps. In the first step the source speech gets transformed into text by using a Speech-to-Text (STT) model. The sec- ond step consist of translating the text to the target lan- guage. The last step takes in the translated text as input and synthesises speech in the target language using a Text-to-Speech (TTS) model. Because of the concate- nation of these three steps, and the intermediate text representation, there is a loss of information from the original speech which does not get passed on after the first step (Sperber and Paulik, 2020). This information loss mainly consists of spoken language characteristics such as the pitch and energy in the source speech which are key elements for prosody. Prosody is essential for a conversation as it contains, for example, information on the emotional state of the speaker. In this work, we want to address this issue by propos- ing dedicated TTS …

Content truncated to 500 words