NVIDIA NeMo Offline Speech Translation Systems for IWSLT 2023

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 442–448 July 13-14, 2023 c⃝2023 Association for Computational Linguistics NVIDIA NeMo Offline Speech Translation Systems for IWSLT 2023 Oleksii Hrinchuk*1, Vladimir Bataev1,2, Evelina Bakhturina1, Boris Ginsburg1 1NVIDIA, Santa Clara, CA 2University of London, London, UK Abstract This paper provides an overview of NVIDIA NeMo’s speech translation systems for the IWSLT 2023 Offline Speech Translation Task. This year, we focused on end-to-end system which capitalizes on pre-trained models and synthetic data to mitigate the problem of di- rect speech translation data scarcity. When trained on IWSLT 2022 constrained data, our best En→De end-to-end model achieves the average score of 31 BLEU on 7 test sets from IWSLT 2010-2020 which improves over our last year cascade (28.4) and end-to-end (25.7) submissions. When trained on IWSLT 2023 constrained data, the average score drops to 29.5 BLEU. 1 Introduction We participate in the IWSLT 2023 Offline Speech Translation Task (Agarwal et al., 2023) for English→German, English→Chinese, and English→Japanese. This year, we focus on an end- to-end model, which directly translates English audio into text in other languages. In contrast to automatic speech recognition (ASR) and text-to-text neural machine translation (NMT), the data for direct speech translation (ST) is scarce and expensive. Thus, to train a high- quality end-to-end ST model, we heavily rely on a number of auxiliary models for which the amount of available data is enough. Specifically, we train the following models: • ASR model with FastConformer- RNNT (Rekesh et al., 2023) architecture trained on all allowed data. • NMT model with Transformer encoder- decoder architecture trained on all allowed bitext and in-domain fine-tuned on TED talks. *Correspondence to: ohrinchuk@nvidia.com • Text-to-speech (TTS) model with Fast- Pitch (Ła´ncucki, 2021) architecture trained on the English transcripts of TED talks. • Supervised Hybrid Audio Segmentation (SHAS) model (Tsiamas et al., 2022) trained on TED talks. Our constrained end-to-end ST model consists of a FastConformer encoder and a Transformer decoder. We initialize the encoder with the corre- sponding component from ASR and train our ST model on a mix of speech-to-text and text-to-text data. We replace all ground truth translations (wher- ever available) with synthetic ones generated with the NMT model and voice the English portion of parallel text corpora with TTS. Our systems will be open-sourced as part of NVIDIA NeMo1 framework (Kuchaiev et al., 2019). 2 Data In this section, we describe the datasets used for training (Table 1). For evaluation, we used the de- velopment sets of Must-C v2 (Cattoni et al., 2021), as well as the test sets from past IWSLT competi- tions. We noticed that development data had a large overlap with training data, mostly because of the usage of the same TED talks in different datasets. Thus, we discarded all samples with overlapping transcripts and talk ids. TED talks In the list of allowed data, there are several datasets comprised of TED talks, namely Must-C v1-v3, ST-TED (Jan et al., 2018), and TED- LIUM v3 (Hernandez …

Content truncated to 500 words