CMU’s IWSLT 2022 Dialect Speech Translation System

Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 298 - 307 May 26-27, 2022 c⃝2022 Association for Computational Linguistics CMU’s IWSLT 2022 Dialect Speech Translation System Brian Yan1 Patrick Fernandes1,2 Siddharth Dalmia1 Jiatong Shi1 Yifan Peng3 Dan Berrebbi1 Xinyi Wang1 Graham Neubig1 Shinji Watanabe1,4 1Language Technologies Institute, Carnegie Mellon University, USA 2Instituto Superior Técnico & LUMLIS (Lisbon ELLIS Unit), Portugal 3Electrical and Computer Engineering, Carnegie Mellon University, USA 4Human Language Technology Center of Excellence, Johns Hopkins University, USA {byan, pfernand, sdalmia, jiatongs}@cs.cmu.edu Abstract This paper describes CMU’s submissions to the IWSLT 2022 dialect speech translation (ST) shared task for translating Tunisian-Arabic speech to English text. We use additional paired Modern Standard Arabic data (MSA) to directly improve the speech recognition (ASR) and machine translation (MT) components of our cascaded systems. We also augment the paired ASR data with pseudo translations via sequence-level knowledge distillation from an MT model and use these artificial triplet ST data to improve our end-to-end (E2E) sys- tems. Our E2E models are based on the Multi- Decoder architecture with searchable hidden intermediates. We extend the Multi-Decoder by orienting the speech encoder towards the target language by applying ST supervision as hierarchical connectionist temporal classifica- tion (CTC) multi-task. During inference, we apply joint decoding of the ST CTC and ST autoregressive decoder branches of our modi- fied Multi-Decoder. Finally, we apply ROVER voting, posterior combination, and minimum bayes-risk decoding with combined N-best lists to ensemble our various cascaded and E2E sys- tems. Our best systems reached 20.8 and 19.5 BLEU on test2 (blind) and test1 respectively. Without any additional MSA data, we reached 20.4 and 19.2 on the same test sets. 1 Introduction In this paper, we present CMU’s Tunisian-Arabic to English ST systems submitted to the IWSLT 2022 dialectal ST track (Anastasopoulos et al., 2022). One of our goals is to investigate dialectal transfer from large MSA ASR and MT corpora to improve Tunisian-Arabic ST performance. We also view this task as setting for extending the sequence-level knowledge distillation (SeqKD) (Kim and Rush, 2016), E2E Multi-Decoder architecture (Dalmia et al., 2021), and system combination methods in our IWSLT 2021 offline ST systems (Inaguma et al., 2021b). In particular, our contributions are the following: 1. Dialectal transfer from large paired MSA cor- pora to improve ASR and MT systems (§3.1) 2. MT SeqKD on MSA ASR data for artificial ST triplets to improve E2E ST systems (§3.2.2) 3. Multi-Decoder with hierarchical CTC training for target-oriented speech encodings (§3.2.3) 4. Multi-Decoder with CTC beam search hypothe- sis re-scoring during ST inference (§3.2.4) 5. Multi-Decoder with surface and posterior-level guidance from external models (§3.3.1) 6. Joint minimum bayes-risk decoding as an en- sembling method (§3.3.2) Results on the blind test set, test2, and ablations on the provided test set, test1, demonstrate the overall efficacy of our systems and the relative contribu- tions of the aforementioned techniques (§5). 2 Task Description and Data Preparation The Arabic language is not a monolith. Of its esti- mated …

Content truncated to 500 words