Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4252–4261 August 1–6, 2021. ©2021 Association for Computational Linguistics 4252 Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task Yun Tang, Juan Pino, Xian Li, Changhan Wang, Dmitriy Genzel Facebook AI {yuntang,juancarabina,xianl,changhan,dgenzel}@fb.com Abstract Pretraining and multitask learning are widely used to improve the speech to text translation performance. In this study, we are interested in training a speech to text translation model along with an auxiliary text to text translation task. We conduct a detailed analysis to un- derstand the impact of the auxiliary task on the primary task within the multitask learning framework. Our analysis confirms that multi- task learning tends to generate similar decoder representations from different modalities and preserve more information from the pretrained text translation modules. We observe mini- mal negative transfer effect between the two tasks and sharing more parameters is helpful to transfer knowledge from the text task to the speech task. The analysis also reveals that the modality representation difference at the top decoder layers is still not negligible, and those layers are critical for the transla- tion quality. Inspired by these findings, we propose three methods to improve translation quality. First, a parameter sharing and ini- tialization strategy is proposed to enhance in- formation sharing between the tasks. Second, a novel attention-based regularization is pro- posed for the encoders and pulls the represen- tations from different modalities closer. Third, an online knowledge distillation is proposed to enhance the knowledge transfer from the text to the speech task. Our experiments show that the proposed approach improves transla- tion performance by more than 2 BLEU over a strong baseline and achieves state-of-the- art results on the MUST-C English-German, English-French and English-Spanish language pairs. 1 Introduction End-to-end methods have achieved significant progress in speech to text translation (ST) and even surpassed the traditional pipeline-based methods in some applications (Niehues et al., 2019; Salesky and Black, 2020). However, the success of end- to-end methods relies on large amounts of training data, which is quite expensive to obtain and rela- tively small in practice. Building ST systems from pretrained models with multitask learning (MTL) is widely used to overcome the limited training data issue (Weiss et al., 2017; Anastasopoulos and Chi- ang, 2018; Bahar et al., 2019; Indurthi et al., 2020; Wang et al., 2020b; Li et al., 2020). Nevertheless, little prior work has been devoted to understanding the interactions between different tasks. Standley et al. (2020) conduct an empirical study on com- puter vision tasks for MTL. They find many “as- sumptions” for MTL may not be held for specific applications. For example, “similar” tasks do not necessarily train better together. In this study, we focus on training the ST model along with an auxiliary text to text machine trans- lation (MT) task. We are interested in the task interactions with different modalities and in im- proving the primary …

Content truncated to 500 words