ESPnet How2 Speech Translation System for IWSLT 2019: Pre-training, Knowledge Distillation, and Going Deeper

ESPnet-ST IWSLT 2021 Offline Speech Translation System Hirofumi Inaguma1∗ Brian Yan2∗ Siddharth Dalmia2 Pengcheng Guo3 Jiatong Shi4 Kevin Duh4 Shinji Watanabe2,4 1Kyoto University, Japan 2Carnegie Mellon University, USA 3Northwestern Polytechnical University, China 4Johns Hopkins University, USA inaguma@sap.ist.i.kyoto-u.ac.jp byan@cs.cmu.edu Abstract This paper describes the ESPnet-ST group’s IWSLT 2021 submission in the offline speech translation track. This year we made various efforts on training data, architecture, and au- dio segmentation. On the data side, we inves- tigated sequence-level knowledge distillation (SeqKD) for end-to-end (E2E) speech transla- tion. Specifically, we used multi-referenced SeqKD from multiple teachers trained on dif- ferent amounts of bitext. On the architecture side, we adopted the Conformer encoder and the Multi-Decoder architecture, which equips dedicated decoders for speech recognition and translation tasks in a unified encoder-decoder model and enables search in both source and target language spaces during inference. We also significantly improved audio segmenta- tion by using the pyannote.audio toolkit and merging multiple short segments for long context modeling. Experimental evaluations showed that each of them contributed to large improvements in translation performance. Our best E2E system combined all the above tech- niques with model ensembling and achieved 31.4 BLEU on the 2-ref of tst2021 and 21.2 BLEU and 19.3 BLEU on the two single refer- ences of tst2021. 1 Introduction This paper presents the ESPnet-ST group’s English→German speech translation (ST) system submitted to the IWSLT 2021 offline speech trans- lation track. ESPnet (Watanabe et al., 2018) has been widely used for many speech applica- tions; automatic speech recognition (ASR), text- to-speech (Hayashi et al., 2020), speech transla- tion (Inaguma et al., 2020), machine translation (MT), and speech separation/enhancement (Li et al., 2021). The purpose of this submission is not only to show the recent progress on ST researches, but ∗*Equal contribution also to encourage future research by building strong systems along with the open-sourced project. This year we focused on (1) sequence-level knowledge distillation (SeqKD) (Kim and Rush, 2016), (2) Conformer encoder (Gulati et al., 2020), (3) Multi-Decoder architecture (Dalmia et al., 2021), (4) model ensembling, and (5) better seg- mentation with a neural network-based voice activ- ity (VAD) system (Bredin et al., 2020) and a novel algorithm to merge multiple short segments for long context modeling. Our primary focus was E2E models, although we also compared them with cascade systems with our best effort. All experiments were conducted with the ESPnet-ST toolkit (Inaguma et al., 2020), and the recipe is pub- licly available at https://github.com/espnet/ espnet/tree/master/egs/iwslt21. 2 Data preparation In this section, we describe data preparation for each task. The corpus statistics are listed in Table 1. We removed the off-limit talks fol- lowing previous evaluation campaigns1. To fit the GPU memory, we excluded utterances hav- ing more than 3000 speech frames or more than 400 characters. All sentences were tokenized with the tokenizer.perl script in the Moses toolkit (Koehn et al., 2007). 2.1 ASR We used Must-C (Di Gangi et al., 2019), Must-C v22, ST-TED (Jan et al., 2018), Librispeech (Panay- otov et al., 2015), …

Content truncated to 500 words