The HW-TSC’s Simultaneous Speech-to-Speech Translation System for IWSLT 2023 Evaluation

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 383–388 July 13-14, 2023 c⃝2023 Association for Computational Linguistics The HW-TSC’s Simultaneous Speech-to-Speech Translation system for IWSLT 2023 evaluation Hengchao Shang, Zhiqiang Rao, Zongyao Li, Jiaxin GUO, Zhanglin Wu, Minghan Wang, Daimeng Wei, Shaojun Li, Zhengzhe Yu, Xiaoyu Chen, Lizhi Lei, Hao Yang {shanghengchao, raozhiqiang, lizongyao, guojiaxin1, wuzhanglin2, wangminghan, weidaimeng, lishaojun18, yuzhengzhe, chenxiaoyu35, leilizhi, yanghao30}@huawei.com Abstract In this paper, we present our submission to the IWSLT 2023 (Agarwal et al., 2023) Simultane- ous Speech-to-Speech Translation competition. Our participation involves three language direc- tions: English-German, English-Chinese, and English-Japanese. Our solution is a cascaded incremental decoding system, consisting of an ASR model, an MT model, and a TTS model. By adopting the strategies used in the Speech- to-Text track, we have managed to generate a more confident target text for each audio seg- ment input, which can guide the next MT in- cremental decoding process. Additionally, we have integrated the TTS model to seamlessly reproduce audio files from the translation hy- pothesis. To enhance the effectiveness of our experiment, we have utilized a range of meth- ods to reduce error conditions in the TTS input text and improve the smoothness of the TTS output audio. 1 Introduction This paper describes the HW-TSC‘s submission to the Simultaneous Speech-to-Speech Translation (SimulS2S) task at IWSLT 2023 (Agarwal et al., 2023). Simultaneous speech-to-speech translation (SimulS2S) is currently being researched using aCascade systems. These systems typically involve streaming Automatic Speech Recognition (ASR) module, a streaming Text-to-Text ma- chine translation (MT) module, and an offline Text-to-Speech(TTS) module, with the option of incorporating additional correction modules. Although integrating these modules can be complex, training each module with sufficient data resources can prove to be worthwhile. Our study adopts a comprehensive approach that utilizes several key components to build a strong system. We incorporate a formidable offline ASR model, a robust offline MT model, and a pre-trained TTS model as the foundation for our system. More- over, we introduce a refined onlinization technique based on the approach developed by (Polák et al., 2022), which seamlessly integrates into the cascade system. Offline TTS models often produce a blank sound at the end of a sentence. As a result, when generat- ing audio results in the simultaneous interpreting mode, it can lead to blank tones between clips, caus- ing the final audio to lack smoothness. To address this issue, we have developed several strategies aimed at mitigating this problem in our work. 2 Related Methods 2.1 ASR In our cascade system, we have incorporated the U2 (Wu et al., 2021) as the ASR module. This framework has the flexibility to be implemented on standard Transformer or Conformer architectures and can perform both streaming and non-streaming ASR. One of the major advantages of U2 over other offline autoregressive ASR models is its ability to support streaming through dynamic chunk training and decoding with a CTC decoder on top of the encoder. Additionally, U2 includes a standard au- toregressive attention decoder …

Content truncated to 500 words