The NPU-MSXF Speech-to-Speech Translation System for IWSLT 2023 Speech-to-Speech Translation Task

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 311–320 July 13-14, 2023 c⃝2023 Association for Computational Linguistics The NPU-MSXF Speech-to-Speech Translation System for IWSLT 2023 Speech-to-Speech Translation Task Kun Song1, Yi Lei1, Peikun Chen1, Yiqing Cao2, Kun Wei1, Yongmao Zhang1, Lei Xie1∗, Ning Jiang3, Guoqing Zhao3 1Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer Science, Northwestern Polytechnical University, China 2Department of Computer Science and Technology, Nanjing University, China 3MaShang Consumer Finance Co., Ltd, China Abstract This paper describes the NPU-MSXF system for the IWSLT 2023 speech-to-speech trans- lation (S2ST) task which aims to translate from English speech of multi-source to Chi- nese speech. The system is built in a cascaded manner consisting of automatic speech recog- nition (ASR), machine translation (MT), and text-to-speech (TTS). We make tremendous ef- forts to handle the challenging multi-source input. Specifically, to improve the robustness to multi-source speech input, we adopt various data augmentation strategies and a ROVER- based score fusion on multiple ASR model outputs. To better handle the noisy ASR tran- scripts, we introduce a three-stage fine-tuning strategy to improve translation accuracy. Fi- nally, we build a TTS model with high nat- uralness and sound quality, which leverages a two-stage framework, using network bottleneck features as a robust intermediate representation for speaker timbre and linguistic content disen- tanglement. Based on the two-stage framework, pre-trained speaker embedding is leveraged as a condition to transfer the speaker timbre in the source English speech to the translated Chinese speech. Experimental results show that our system has high translation accuracy, speech naturalness, sound quality, and speaker simi- larity. Moreover, it shows good robustness to multi-source data. 1 Introduction In this paper, we describe NPU-MSXF team’s cas- caded speech-to-speech translation (S2ST) system submitted to the speech-to-speech (S2S) track1 of the IWSLT 2023 evaluation campaign. The S2S track aims to build an offline system that realizes speech-to-speech translation from English to Chi- nese. Particularly, the track allows the use of large- scale data, including the data provided in this track as well as all training data from the offline track2 on ∗Lei Xie is the corresponding author. 1https://iwslt.org/2023/s2s 2https://iwslt.org/2023/offline speech-to-text translation task. Challengingly, the test set contains multi-source speech data, covering a variety of acoustic conditions and speaking styles, designed to examine the robustness of the S2ST system. Moreover, speaker identities conveyed in the diverse multi-source speech test data are unseen during training, which is called zero-shot S2ST and better meets the demands of real-world applica- tions. Current mainstream S2ST models usually in- clude cascaded and end-to-end systems. Cascaded S2ST systems, widely used in the speech-to-speech translation task (Nakamura et al., 2006), usually contain three modules, i.e. automatic speech recog- nition (ASR), machine translation (MT), and text- to-speech (TTS). Meanwhile, end-to-end (E2E) S2ST systems (Jia et al., 2019; Lee et al., 2022) have recently come to the stage by integrating the above modules into a unified model for directly syn- thesizing target language speech translated from the source language. E2E S2ST systems …

Content truncated to 500 words