On Knowledge Distillation for Translating Erroneous Speech Transcriptions

Proceedings of the 18th International Conference on Spoken Language Translation, pages 198–205 Bangkok, Thailand (Online), August 5–6, 2021. ©2021 Association for Computational Linguistics 198 On Knowledge Distillation for Translating Erroneous Speech Transcriptions Ryo Fukuda1, Katsuhito Sudoh1,2, and Satoshi Nakamura1,2 1Nara Institute of Science and Technology, Japan 2AIP, RIKEN, Japan {fukuda.ryo.fo3, sudoh, s-nakamura}@is.naist.jp Abstract Recent studies argue that knowledge distilla- tion is promising for speech translation (ST) using end-to-end models. In this work, we in- vestigate the effect of knowledge distillation with a cascade ST using automatic speech recognition (ASR) and machine translation (MT) models. We distill knowledge from a teacher model based on human transcripts to a student model based on erroneous transcrip- tions. Our experimental results demonstrated that knowledge distillation is beneficial for a cascade ST. Further investigation that com- bined knowledge distillation and fine-tuning revealed that the combination consistently im- proved two language pairs: English-Italian and Spanish-English. 1 Introduction Speech translation (ST) converts utterances in a source language into text in another language. Con- ventional ST systems called cascade or pipeline ST consist of two components: automatic speech recognition (ASR) and machine translation (MT). In the cascade ST, the error propagation from ASR to MT seriously degrades the ST performance. On the other hand, a new ST system called end-to-end or direct ST uses a single model to directly trans- late the source language speech into target language text (B´erard et al., 2016). Such an end-to-end ap- proach is a new paradigm in ST and is attracting much research attention. However, a naive end-to- end ST without additional training, such as ASR tasks, remains inferior to a cascade ST (Liu et al., 2018; Salesky and Black, 2020). Additionally, it requires parallel data of the source language speech and the target language text, which cannot be ob- tained easily in practice. Recent ST studies have incorporated the tech- niques of cascade ST to end-to-end STs. Multi- task training with an ASR subtask has been used successfully in end-to-end ST (Weiss et al., 2017; Anastasopoulos and Chiang, 2018; Sperber et al., 2019). Initializing an end-to-end ST with a pre- trained ASR or MT has also become a common approach (B´erard et al., 2018; Bansal et al., 2019; Inaguma et al., 2020; Wang et al., 2020; Bahar et al., 2021). In this work, we focus on the cascade approach due to its performance advantage against end-to- end STs. Another reason is that cascade ST models can be incorporated into end-to-end STs, as shown in previous studies. During the training of an MT model for a cas- cade ST, we can use clean human transcripts for the source language speech as input. However, since the MT in a cascade ST always receives ASR out- put during inferences, ASR errors should be propa- gated to the MT model to cause translation errors. What if we use erroneous speech transcriptions by ASR for training? That approach means the MT model is trained to translate erroneous transcrip- tions into correct text, which would not generally …

Content truncated to 500 words