NAIST Simultaneous Speech-to-speech Translation System for IWSLT 2023

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 330–340 July 13-14, 2023 c⃝2023 Association for Computational Linguistics NAIST Simultaneous Speech Translation System for IWSLT 2023 Ryo Fukuda† Yuta Nishikawa† Yasumasa Kano† Yuka Ko† Tomoya Yanagita† Kosuke Doi† Mana Makinae† Sakriani Sakti‡† Katsuhito Sudoh† Satoshi Nakamura† †Nara Institute of Science and Technology, Japan ‡Japan Advanced Institute of Science and Technology, Japan fukuda.ryo.fo3@is.naist.jp Abstract This paper describes NAIST’s submission to the IWSLT 2023 Simultaneous Speech Translation task: English-to-{German, Japanese, Chinese} speech-to-text translation and English-to-Japanese speech-to-speech translation. Our speech-to-text system uses an end-to-end multilingual speech translation model based on large-scale pre-trained speech and text models. We add Inter-connections into the model to incorporate the outputs from intermediate layers of the pre-trained speech model and augment prefix-to-prefix text data using Bilingual Prefix Alignment to enhance the simultaneity of the offline speech translation model. Our speech-to-speech system employs an incremental text-to-speech module that consists of a Japanese pronuncia- tion estimation model, an acoustic model, and a neural vocoder. 1 Introduction This paper presents NAIST’s simultaneous speech translation (SimulST) systems for the IWSLT 2023 English-to-{German, Japanese, Chinese} speech-to-text track and the English-to-Japanese speech-to-speech track (Agarwal et al., 2023). Many previous studies on end-to-end SimulST have focused on training methodologies and ar- chitectures specialized for the simultaneous sce- nario. However, such a specialized system setup for SimulST is not trivial and increases the dif- ficulty of the system development and the com- putational complexity. One recent approach to SimulST systems is to use an offline speech trans- lation (ST) model for prefix-to-prefix translation required in SimulST. In last year’s IWSLT Eval- uation Campaign (Anastasopoulos et al., 2022), Polák et al. (2022) demonstrated superior results using such multilingual offline ST models. In our last year’s systems (Fukuda et al., 2022), we used an offline model fine-tuned for SimulST with data augmentation based on Bilingual Prefix Align- ment (Kano et al., 2022). In this year, we use an end-to-end multilin- gual offline ST model based on large-scale pre- trained speech and text models for the speech- to-text track, following Polák et al. (2022). We used Hidden-Unit BERT (HuBERT) (Hsu et al., 2021) as the speech encoder fine-tuned using En- glish automatic speech recognition (ASR) data and mBART50 (Tang et al., 2020) as the text decoder fine-tuned using a multilingual machine translation data. We prepare the multilingual ST model in the following steps: 1. Initialize the model with the parameters of HuBERT and mBART50 models and add Inter-connections between the intermediate layer of the speech encoder and the text de- coder. 2. Train the model using multilingual ST cor- pora. 3. Fine-tune the model using bilingual prefix pairs in English-to-{German, Japanese, Chi- nese} extracted using Bilingual Prefix Align- ment. We use a SimulST policy called local agreement (Liu et al., 2020) that finds the longest common prefixes among successive decoding steps. For the English-to-Japanese speech-to-speech track, we developed a cascade of the SimulST above and an incremental text-to-speech module using a pro- nunciation estimation model, …

Content truncated to 500 words