The NiuTrans End-to-End Speech Translation System for IWSLT23 English-to-Chinese Offline Task

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 211–218 July 13-14, 2023 c⃝2023 Association for Computational Linguistics The NiuTrans End-to-End Speech Translation System for IWSLT23 English-to-Chinese Offline Task Yuchen Han1∗, Xiaoqian Liu1∗, Hao Chen1, Yuhao Zhang1, Chen Xu1, Tong Xiao1,2, Jingbo Zhu1,2 1School of Computer Science and Engineering, Northeastern University, Shenyang, China 2NiuTrans Research, Shenyang, China {hanyuchen114,yoohao.zhang}@gmail.com,methanechen@126.com {liuxiaoqian0319,xuchennlp}@outlook.com {xiaotong,zhujingbo}@mail.neu.edu.cn Abstract This paper describes the NiuTrans end-to-end speech translation system submitted for the IWSLT 2023 English-to-Chinese offline task. Our speech translation models are composed of pre-trained ASR and MT models under the stacked acoustic and textual encoding frame- work. Several pre-trained models with diverse architectures and input representations (e.g., log Mel-filterbank and waveform) were utilized. We proposed an iterative data augmentation method to iteratively improve the performance of the MT models and generate the pseudo ST data through MT systems. We then trained ST models with different structures and data set- tings to enhance ensemble performance. Exper- imental results demonstrate that our NiuTrans system achieved a BLEU score of 29.22 on the MuST-C En-Zh tst-COMMON set, outper- forming the previous year’s submission by 0.12 BLEU despite using less MT training data. 1 Introduction End-to-end speech translation (E2E ST) directly translate speech in the source language into text in the target language without generating an interme- diate representation, which has gained significant attention in recent years due to several advantages over cascade methods, including low latency and the ability to avoid error propagation (Berard et al., 2016; Weiss et al., 2017). In this paper, we describe our NiuTrans E2E ST system that participated in the IWSLT23 English-to-Chinese offline track, the overview of our system is shown in Fig 1. To improve the performance of our system, we aim to maximize the diversity of our ensemble of E2E ST models. Our E2E ST models are based on the stacked acoustic and textual encoding (SATE) method (Xu et al., 2021a), which is a framework to make the best of pre-trained automatic speech recognition (ASR) and machine translation (MT) *Authors contributed equally. FBank/ wav pseudo ST pseudo MT ASR FBank/wav labeled ASR labeled MT labeled ST MT (ensemble) ST iteratively decode initialize train Figure 1: Overview of our system. components. Using this framework, we explore multiple architectures of pre-trained ASR and MT models with varying numbers of parameters and input representations such as FBank features or waveform data. Pseudo data is a crucial component of E2E ST, often generated by ensemble MT systems (Gaido et al., 2020). This year, we focused more on the per- formance of MT models and developed an Iterative Data Augmentation method to leverage text data from all corpora, improving the MT models and enabling the generation of multiple pseudo data. We then used these multiple pseudo data to train diverse E2E ST models for optimal performance. Our best ST ensemble system includes models with different input representations, architectures, and training corpora, achieving a BLEU score of 29.22 on the MuST-C En-Zh tst-COMMON set. The remainder …

Content truncated to 500 words