On the Copying Problem of Unsupervised NMT: A Training Schedule with a Language Discriminator Loss

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 89–101 July 13-14, 2023 c⃝2023 Association for Computational Linguistics Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic Transcripts Rebekka Hubert∗ Computational Linguistics Heidelberg University, Germany hubert@cl.uni-heidelberg.de Artem Sokolov Google Research Berlin, Germany artemsok@google.com Stefan Riezler Computational Linguistics & IWR Heidelberg University, Germany riezler@cl.uni-heidelberg.de Abstract End-to-end automatic speech translation (AST) relies on data that combines audio inputs with text translation outputs. Previous work used ex- isting large parallel corpora of transcriptions and translations in a knowledge distillation (KD) setup to distill a neural machine transla- tion (NMT) into an AST student model. While KD allows using larger pretrained models, the reliance of previous KD approaches on manual audio transcripts in the data pipeline restricts the applicability of this framework to AST. We present an imitation learning approach where a teacher NMT system corrects the errors of an AST student without relying on manual tran- scripts. We show that the NMT teacher can recover from errors in automatic transcriptions and is able to correct erroneous translations of the AST student, leading to improvements of about 4 BLEU points over the standard AST end-to-end baseline on the English-German CoVoST-2 and MuST-C datasets, respectively. Code and data are publicly available.1 1 Introduction The success of data-hungry end-to-end automatic speech translation (AST) depends on large amounts of data that consist of speech inputs and corre- sponding translations. One way to overcome the data scarcity issue is a knowledge distillation (KD) setup where a neural machine translation (NMT) expert (also called oracle) is distilled into an AST student model (Liu et al., 2019; Gaido et al., 2020). The focus of our work is the question of whether the requirement of high-quality source language tran- scripts, as in previous applications of KD to AST, can be relaxed in order to enable a wider applicabil- ity of this setup to AST scenarios where no manual source transcripts are available. Examples for such ∗All work was done at Heidelberg University. 1https://github.com/HubReb/imitkd_ast/ releases/tag/v1.1 scenarios are low-resource settings (e.g., for lan- guages without written form for which mostly only audio-translation data are available), or settings where one of the main uses of source transcripts in AST — pre-training the AST encoder from an automatic speech recognition (ASR) system— is replaced by a large-scale pre-trained ASR system (which itself is trained on hundreds of thousands hours of speech, but the original training transcripts are not available (Radford et al., 2022; Zhang et al., 2022b)). Relaxing the dependence of pre-training AST encoders on manual transcripts has recently been studied by Zhang et al. (2022a). Our focus is instead to investigate the influence of manual versus synthetic transcripts as input to the student model in an imitation learning (IL) approach (Lin et al., 2020; Hormann and Sokolov, 2021), and to lift this scenario to AST. To our knowledge, this has not been attempted before. We present a proof-of- concept experiment where we train an ASR model on a few …

Content truncated to 500 words