THE IWSLT 2021 BUT SPEECH TRANSLATION SYSTEMS

Proceedings of the 18th International Conference on Spoken Language Translation, pages 75–83 Bangkok, Thailand (Online), August 5–6, 2021. ©2021 Association for Computational Linguistics 75 The IWSLT 2021 BUT Speech Translation Systems Hari Krishna Vydana, Martin Karafiát, Lukáš Burget, “Honza” ˇCernocký Brno University of Technology, Faculty of Information Technology, Speech@FIT, Czechia harivydana@gmail.com Abstract The paper describes BUT’s English to Ger- man offline speech translation (ST) systems developed for IWSLT2021. They are based on jointly trained Automatic Speech Recognition- Machine Translation models. Their perfor- mances is evaluated on MustC-Common test set. In this work, we study their efficiency from the perspective of having a large amount of separate ASR training data and MT train- ing data, and a smaller amount of speech- translation training data. Large amounts of ASR and MT training data are utilized for pre- training the ASR and MT models. Speech- translation data is used to jointly optimize ASR-MT models by defining an end-to-end differentiable path from speech to translations. For this purpose, we use the internal continu- ous representations from the ASR-decoder as the input to MT module. We show that speech translation can be further improved by training the ASR-decoder jointly with the MT-module using large amount of text-only MT training data. We also show significant improvements by training an ASR module capable of gener- ating punctuated text, rather than leaving the punctuation task to the MT module. 1 Introduction Speech Translation (ST) systems are intended to generate text in target language from the audio in source language. The conventional ST systems are cascade ones, including (in the most popular form) three blocks i.e., an ASR, punctuation/segmentation module and an MT model (Ngoc-Quan Pham, 2019; Pham et al., 2020b; Jan et al., 2019; Ansari et al., 2020). Both Automatic Speech Recognition system (ASR) and Machine Translation (MT) models are indepen- dently trained, and the MT model processes the ASR output text (ASR hypotheses) to generate translations. In a cascade system, the advance- ments in ASR and MT can be directly extended to ST. These models can also leverage on the avail- ability of large ASR and MT data-sets, and some of the state-of-the art ST systems are still cascade ones. Recently, End-to-End ST systems have become widely popular. An End-to-End ST can directly generate text in target language from the audio in source language. These models are simpler in structure and they are more suitable for op- erating in streaming fashion. Most End-to-End speech translation systems are variants of encoder- decoder architecture with attention models (Bah- danau et al., 2015; Di Gangi et al., 2019; Zhao et al., 2020). This category includes the popu- lar Transformer models, which have been adapted for training End-to-End ST in (Di Gangi et al., 2019). In (Inaguma et al., 2020), a better perfor- mance of ST was achieved by initializing the en- coder and decoder modules from pre-trainied ASR and MT systems, respectively. Very-deep trans- former models have been trained with stochastic depth for training End-to-End ST models in (Pham et …

Content truncated to 500 words