CMU’s IWSLT 2023 Simultaneous Speech Translation System

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 235–240 July 13-14, 2023 c⃝2023 Association for Computational Linguistics CMU’s IWSLT 2023 Simultaneous Speech Translation System Brian Yan*1 Jiatong Shi*1 Soumi Maiti1 William Chen1 Xinjian Li1 Yifan Peng2 Siddhant Arora1 Shinji Watanabe1,3 1Language Technologies Institute, Carnegie Mellon University, USA 2Electrical and Computer Engineering, Carnegie Mellon University, USA 3Human Language Technology Center of Excellence, Johns Hopkins University, USA {byan, jiatongs}@cs.cmu.edu Abstract This paper describes CMU’s submission to the IWSLT 2023 simultaneous speech transla- tion shared task for translating English speech to both German text and speech in a stream- ing fashion. We first build offline speech-to- text (ST) models using the joint CTC/attention framework. These models also use WavLM front-end features and mBART decoder initial- ization. We adapt our offline ST models for simultaneous speech-to-text translation (SST) by 1) incrementally encoding chunks of input speech, re-computing encoder states for each new chunk and 2) incrementally decoding out- put text, pruning beam search hypotheses to 1- best after processing each chunk. We then build text-to-speech (TTS) models using the VITS framework and achieve simultaneous speech- to-speech translation (SS2ST) by cascading our SST and TTS models. 1 Introduction In this paper, we present CMU’s English to Ger- man simultaneous speech translation systems. Our IWSLT 2023 (Agarwal et al., 2023) shared task submission consists of both simultaneous speech- to-text (SST) and simultaneous speech-to-speech (SS2ST) systems. Our general strategy is to first build large-scale offline speech translation (ST) models which leverage unpaired speech data, ASR data, and ST data. We then adapt these offline models for simultaneous inference. Finally, we use a text-to-speech model to achieve SS2ST in a cascaded manner. In particular, our system consists of: 1. Offline ST using joint CTC/attention with self- supervised speech/text representations (§3.1) 2. Offline-to-online adaptation via chunk-based en- coding and incremental beam search (§3.2) 3. Simultaneous S2ST by feeding incremental text outputs to a text-to-speech model (§3.3) 2 Task Description The IWSLT 2023 simultaneous speech translation track1 is a shared task for streaming speech-to- text and speech-to-speech translation of TED talks. This track mandates that systems do not perform re-translation, meaning that the streaming outputs cannot be edited after the system receives more input audio. Systems are required to meet a par- ticular latency regime: SST systems must have <2 seconds average lagging (AL) and SS2ST systems must have <2.5 seconds start offset (SO) (Ma et al., 2020). Of the allowed training data, we selected a sub- set of in-domain data to train our ASR and ST models: for ASR we use TEDLIUM v1 and v2 (Zhou et al., 2020) and for ST we used MuST- C v2 (Di Gangi et al., 2019). We also use a set of cross-domain data to train our MT and TTS models due to the lack of in-domain data: for MT we use Europarl, NewsCommentary, Open- Subtitles, TED2020, Tatoeba, and ELRC-CORDIS News (Tiedemann et al., 2020). For TTS we use CommonVoice (Ardila et al., 2020). The following section describes …

Content truncated to 500 words