Improving Low Resource Speech Translation with Data Augmentation and Ensemble Strategies

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 241–250 July 13-14, 2023 c⃝2023 Association for Computational Linguistics Improving Low Resource Speech Translation with Data Augmentation and Ensemble Strategies Akshaya Vishnu Kudlu Shanbhogue Ran Xue∗ Soumya Saha∗ Daniel Yue Zhang∗ Ashwinkumar Ganesan∗ Amazon Alexa AI {ashanbho,ranxue,soumyasa,dyz,gashwink}@amazon.com Abstract This paper describes the speech translation system submitted as part of the IWSLT 2023 shared task on low resource speech transla- tion. The low resource task aids in building models for language pairs where the training corpus is limited. In this paper, we focus on two language pairs, namely, Tamasheq-French (Tmh→Fra) and Marathi-Hindi (Mr→Hi) and implement a speech translation system that is unconstrained. We evaluate three strategies in our system: (a) Data augmentation where we perform different operations on audio as well as text samples, (b) an ensemble model that integrates a set of models trained using a combination of augmentation strategies, and (c) post-processing techniques where we explore the use of large language models (LLMs) to improve the quality of sentences that are gen- erated. Experiments show how data augmenta- tion can relatively improve the BLEU score by 5.2% over the baseline system for Tmh→Fra while an ensemble model further improves per- formance by 17% for Tmh→Fra and 23% for Mr→Hi task. 1 Introduction Speech translation (ST) systems have multiple ap- plications. They can be utilized in a wide range of scenarios such as closed captioning in different languages while watching videos or even as a real- time assistant that translates speeches to live audi- ences. One persistent challenge for speech trans- lation systems continues to be performing transla- tions for low resource language pairs.1 The IWSLT 2023 (Agarwal et al., 2023) shared task for low resource speech translation targets 8 language pairs that include Tunisian Arabic (Aeg) to English (En), Irish (Ga) to English (En), Marathi (Mr) to Hindi (Hi), Maltese (Mlt) to English (En), Pashto (Pus) to French (Fr), Tamasheq (Tmh) to French (Fr), ∗ These authors contributed equally to this work. 1https://iwslt.org/2023/low-resource and Quechua (Que) to Spanish (Es). This paper outlines a low resource speech translation system (from the Amazon Alexa AI team) for 2 language pairs, namely, Tamasheq-French (Tmh→Fra) and Marathi-Hindi (Mr→Hi). Depending on the type of output that is gener- ated the end-to-end speech translation task has two formats: (a) Speech-to-text (S2T), and (b) Speech- to-Speech (S2S). There are two types of ST sys- tems. The first is a cascaded system where speech recognition and language translation are decou- pled.2 The second is an end-to-end (E2E) model that combines both audio processing and language translation. We design and evaluate an E2E model in this paper. In the past, various approaches have been pro- posed to build E2E low resource speech translation models. Bansal et al. (2018) designs an initial sys- tem that is an encoder-decoder architecture that integrates a convolutional neural network (CNN) and recurrent neural network (RNN). Stoian et al. (2020) try to improve ST models for low resource languages by pretraininig …

Content truncated to 500 words