Mutual-Learning Improves End-to-End Speech Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3989–3994 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3989 Mutual-Learning Improves End-to-End Speech Translation Jiawei Zhao1∗, Wei Luo3, Boxing Chen3, Andrew Gilman1,2 1School of Natural and Computational Sciences, Massey University, New Zealand 2PlantTech Research Institute, New Zealand 3Alibaba Inc. j.zhao2@massey.ac.nz andrew@pri.co.nz {muzhuo.lw, boxing.cbx}@alibaba-inc.com Abstract A currently popular research area in end-to- end speech translation is the use of knowledge distillation from a machine translation (MT) task to improve the speech translation (ST) task. However, such scenario obviously allows only a one way transfer, limiting the overall effec- tiveness of the approach by the performance of the pre-trained teacher model. Therefore, we pose that in this respect knowledge distillation- based approaches are sub-optimal. We propose an alternative—a trainable mutual-learning sce- nario, where the MT and ST models are collab- oratively trained and are considered as peers, rather than teacher/student. This allows us to improve the performance of end-to-end ST more effectively than with a teacher-student paradigm. As a side benefit, performance of the MT model also improves. Experimental results show that in our mutual-learning sce- nario, models can effectively utilise the auxil- iary information from peer models and achieve compelling results on MuST-C datasets. 1 Introduction Speech translation (ST) aims to translate speech sig- nals into a foreign language. It is a multi-modality task, closely related to automatic speech recogni- tion (ASR) and machine translation (MT). ST has a wide range of applications, such as video subti- tling (Saboo and Baumann, 2019), real-time lecture translation (Müller et al., 2016), and protection of endangered languages (Bansal et al., 2017). Despite the recent success in end-to-end (E2E) models, currently such systems still face the issue of labelled training data insufficiency (Sperber and Paulik, 2020). A recent popular advance in E2E ST is the use of knowledge distillation (KD), which can provide an effective paradigm for transfer- ring knowledge from rich-resource to low-resource ∗ Majority of this work was conducted during Jiawei Zhao’s research internship at DAMO Academy, Alibaba. tasks (Liu et al., 2019; Gaido et al., 2020). Un- der such paradigm the MT model is considered a teacher that guides the ST model, considered a student learning from the teacher. We pose that one- way knowledge transfer in a strict teacher-student relationship maybe sub-optimal for the following reasons: 1. Since the MT model is frozen in this one-way knowledge transfer scenario, the success of knowledge transfer and hence the performance of the ST task is constrained by the performance of the pre-trained MT model; 2. There is a modal- ity gap between speech and text inputs of the two models, with speech input also containing inherent speaker variability. Motivated to address the issues mentioned above, we set out to improve ST and MT tasks by training them jointly. Instead of freezing teacher model, we introduce a mutual-learning paradigm, which regards ST and MT models as peers that learn col- laboratively, aiming to iteratively learn and share the knowledge between …

Content truncated to 500 words