Submission of USTC’s System for the IWSLT 2023 - Offline Speech Translation Track

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 194–201 July 13-14, 2023 c⃝2023 Association for Computational Linguistics The USTC’s Offline Speech Translation Systems for IWSLT 2023 Xinyuan Zhou2, Jianwei Cui1, Zhongyi Ye2, Yichi Wang1, Luzhen Xu1, Hanyi Zhang2, Weitai Zhang1,2, Lirong Dai1 1University of Science and Technology of China, Hefei, China 2iFlytek Research, Hefei, China {jwcui,wangyichi,lzxu,zwt2021}@mail.ustc.edu.cn lrdai@ustc.edu.cn {xyzhou15,zyye7,hyzhang56}@iflytek.com Abstract This paper describes the submissions of the research group USTC-NELSLIP to the 2023 IWSLT Offline Speech Translation competi- tion, which involves translating spoken English into written Chinese. We utilize both cascaded models and end-to-end models for this task. To improve the performance of the cascaded mod- els, we introduce Whisper to reduce errors in the intermediate source language text, achiev- ing a significant improvement in ASR recog- nition performance. For end-to-end models, we propose Stacked Acoustic-and-Textual En- coding extension (SATE-ex), which feeds the output of the acoustic decoder into the textual decoder for information fusion and to prevent error propagation. Additionally, we improve the performance of the end-to-end system in translating speech by combining the SATE-ex model with the encoder-decoder model through ensembling. 1 Introduction This paper describes the submission for the IWSLT 2023 Offline Speech Translation task (Agarwal et al., 2023) by National Engineering Laboratory for Speech and Language Information Processing (NELSLIP) at the University of Science and Tech- nology of China. Speech translation (ST) solutions include cas- caded and end-to-end approaches. The cascaded approach combines Automatic Speech Recogni- tion (ASR) and Machine Translation (MT) systems. The ASR system recognizes the source speech as in- termediate text in the source language, and the MT system translates the intermediate text into text in the target language. While the end-to-end approach directly translates the source speech into text in tar- get language, without using source language text as an intermediate representation. Compared with cascaded approaches, the end-to-end paradigm can overcome higher architectural complexity and er- ror propagation (Duong et al., 2016). The Stacked Acoustic-and-Textual Encoding (SATE) (Xu et al., 2021) method combines the acoustic and textual encoders using an adapter module to approach the performance levels of cascaded solutions. Further- more, ST can be improved using large-scale and cross-modal pretraining methods (Radford et al., 2022; Zhang et al., 2022b) such as Whisper (Rad- ford et al., 2022), which leverages large-scale weak supervision, and SpeechUT (Zhang et al., 2022b), which optimizes the alignment of speech and text modalities by hidden units. In this study, we employ a cascaded approach wherein the ASR system is built using the pre- trained Whisper (Radford et al., 2022) to ensure the recognition performance of speech to source language text. Furthermore, the MT systems in the cascaded setup are created using diverse techniques like back translation (Sennrich et al., 2016a), self- training (Kim and Rush, 2016; Liu et al., 2019), domain adaptation and model ensemble. In end-to-end condition, we implement two types of architectures, including encoder-decoder (Le et al., 2021) and Stacked Acoustic-and-Textual Encoding extension (SATE-ex). For the encoder- decoder, we use the …

Content truncated to 500 words