The Kyoto Speech-to-Speech Translation System for IWSLT 2023

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 357–362 July 13-14, 2023 c⃝2023 Association for Computational Linguistics Kyoto Speech-to-Speech Translation System for IWSLT 2023 Zhengdong Yang1 Shuichiro Shimizu1 Zhou Wangjin1 Sheng Li2 Chenhui Chu1 Kyoto University1 National Institute of Information and Communications Technology2 {zd-yang, sshimizu, chu}@nlp.ist.i.kyoto-u.ac.jp zhou@sap.ist.i.kyoto-u.ac.jp sheng.li@nict.go.jp Abstract This paper describes the Kyoto speech-to- speech translation system for IWSLT 2023. Our system is a combination of speech-to-text translation and text-to-speech synthesis. For the speech-to-text translation model, we used the dual-decoder Transformer model. For the text-to-speech synthesis model, we took a cas- cade approach of an acoustic model and a vocoder. 1 Introduction This paper describes the Kyoto speech-to-speech translation system for IWSLT 2023 (Agarwal et al., 2023). Our system is a combination of speech-to- text translation and text-to-speech synthesis. For speech-to-text translation model, we used dual- decoder Transformer model following Le et al. (2020). For text-to-speech synthesis model, we took cascade approach of an acoustic model and a vocoder. We used FastSpeech 2 (Ren et al., 2021) as the acoustic model and HiFi-GAN (Kong et al., 2020) as the vocoder. 2 System Description The speech-to-speech translation system is a com- bination of speech-to-text translation and text-to- speech synthesis. 2.1 Speech-to-Text Translation We adopt the end-to-end speech-to-text translation architecture. The speech-to-text translation model is based on dual-decoder Transfomer (Le et al., 2020). As shown in Figure 1, the model is a Transformer-based model, comprising two de- coders - one for speech-to-text translation (ST) and the other for automatic speech recognition (ASR). The task of ASR and ST can be defined as follows: • For ASR, the input sequence s = [s1, ..., sTs] is a sequence of speech features. The out- put sequence x = [x1, ..., xTx] is the corre- sponding transcription, where Tx indicates the length of the transcription. • For ST, the input sequence s = [s1, ..., sTs] is the same with ASR and the output sequence y = [y1, ..., yTy] is the corresponding transla- tion in target language, where Ty indicates the length of the translation. The model performs the multi-task learning of ASR and ST and the output distributions can be written as Dasr-st = p(x, y|s) = max(Tx,Ty) � t=0 p(xt, yt|x<t, y<t, s) (1) The training objective is a weighted sum of cross- entropy losses for both tasks: Lasr-st = αLasr + (1 − α)Lst (2) Different decoders can exchange information with each other with the interactive attention mech- anism, which refers to replacing attention sub- layers in the standard Transformer decoder with interactive attention sub-layers (Liu et al., 2020). In our models, the replaced sub-layers are the encoder- decoder attention sub-layers. As illustrated in the lower part of Figure 1, an interactive attention sub-layer consists of one main attention sub-layer and a cross-attention sub-layers. The main attention sub-layer is the same as the replaced attention sub-layer. The cross-attention sub-layers receive query Q from the same decoder A and receive key K and value V from …

Content truncated to 500 words