Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1872–1881 June 6–11, 2021. ©2021 Association for Computational Linguistics 1872 Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation Hirofumi Inaguma♣ Tatsuya Kawahara♣ Shinji Watanabe♠ ♣ Kyoto University, Japan ♠ Johns Hopkins University, USA inaguma@sap.ist.i.kyoto-u.ac.jp Abstract A conventional approach to improving the performance of end-to-end speech translation (E2E-ST) models is to leverage the source transcription via pre-training and joint training with automatic speech recognition (ASR) and neural machine translation (NMT) tasks. How- ever, since the input modalities are different, it is difficult to leverage source language text suc- cessfully. In this work, we focus on sequence- level knowledge distillation (SeqKD) from ex- ternal text-based NMT models. To leverage the full potential of the source language infor- mation, we propose backward SeqKD, SeqKD from a target-to-source backward NMT model. To this end, we train a bilingual E2E-ST model to predict paraphrased transcriptions as an aux- iliary task with a single decoder. The para- phrases are generated from the translations in bitext via back-translation. We further propose bidirectional SeqKD in which SeqKD from both forward and backward NMT models is combined. Experimental evaluations on both autoregressive and non-autoregressive models show that SeqKD in each direction consis- tently improves the translation performance, and the effectiveness is complementary regard- less of the model capacity. 1 Introduction End-to-end speech translation (E2E-ST) (Bérard et al., 2016), which aims to convert source speech to text in another language directly, is an active research area. Because direct ST is a more diffi- cult task than automatic speech recognition (ASR) and machine translation (MT), various techniques have been proposed to ease the training process by using source transcription. Examples include pre- training (Bérard et al., 2018; Wang et al., 2020c; Bansal et al., 2019; Wang et al., 2020d), multi-task learning (Weiss et al., 2017; Bérard et al., 2018; Ba- har et al., 2019), knowledge distillation (Liu et al., 2019), meta-learning (Indurthi et al., 2020), two- pass decoding (Anastasopoulos and Chiang, 2018; Sperber et al., 2019), and interactive decoding (Liu et al., 2020; Le et al., 2020). However, as input modalities between ST and MT tasks are different, an auxiliary MT task is not always helpful, espe- cially when additional bitext is not available (Ba- har et al., 2019). Moreover, because monotonic speech-to-transcription alignments encourage the ASR task to see surface-level local information, an auxiliary ASR task helps the E2E-ST model to ex- tract acoustic representations, not semantic ones, from speech. Sequence-level knowledge distillation (Se- qKD) (Kim and Rush, 2016) is another approach to transferring knowledge from one model to an- other. Recent studies have shown that SeqKD has the effect of reducing the complexity of training data and thus eases the training of student models, e.g., non-autoregressive (NAR) models (Gu et al., 2018; Zhou et al., 2019a; Ren et al., 2020). Paraphrasing, which represents text in a differ- ent form but with the same meaning, can also be …

Content truncated to 500 words