Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 3520–3533 Barcelona, Spain (Online), December 8-13, 2020 3520 Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation Hang Le1 Juan Pino2 Changhan Wang2 Jiatao Gu2 Didier Schwab1 Laurent Besacier1 1Univ. Grenoble Alpes, CNRS, LIG 2Facebook AI {hang.le, didier.schwab, laurent.besacier}@univ-grenoble-alpes.fr {juancarabina, changhan, jgu}@fb.com Abstract We introduce dual-decoder Transformer, a new model architecture that jointly performs auto- matic speech recognition (ASR) and multilingual speech translation (ST). Our models are based on the original Transformer architecture (Vaswani et al., 2017) but consist of two decoders, each responsible for one task (ASR or ST). Our major contribution lies in how these decoders interact with each other: one decoder can attend to different information sources from the other via a dual-attention mechanism. We propose two variants of these architectures corresponding to two different levels of dependencies between the decoders, called the parallel and cross dual-decoder Transformers, respectively. Extensive experiments on the MuST-C dataset show that our models outperform the previously-reported highest translation performance in the multilingual settings, and outperform as well bilingual one-to-one results. Furthermore, our parallel models demon- strate no trade-off between ASR and ST compared to the vanilla multi-task architecture. Our code and pre-trained models are available at https://github.com/formiel/speech-translation. 1 Introduction While cascade speech-to-text translation (ST) systems operate in two steps: source language automatic speech recognition (ASR) and source-to-target text machine translation (MT), recent works have at- tempted to build end-to-end ST without using source language transcription during decoding (Bérard et al., 2016; Weiss et al., 2017; Bérard et al., 2018). After two years of extensions to these pioneering works, the last results of the IWSLT 2020 shared task on offline speech translation (Ansari et al., 2020) demonstrate that end-to-end models are now on par (if not better) than their cascade counterparts. Such a finding motivates even more strongly the works on multilingual (one-to-many, many-to-one, many-to- many) ST (Gangi et al., 2019; Inaguma et al., 2019; Wang et al., 2020a) for which end-to-end models are well adapted by design. Moreover, of these two approaches: cascade proposes a very loose integration of ASR and MT (even if lattices or word confusion networks were used between ASR and MT before end-to-end models appeared) while most end-to-end approaches simply ignore ASR subtask, trying to directly translate from source speech to target text. We believe that these are two edge design choices and that a tighter coupling of ASR and MT is desirable for future end-to-end ST applications, in which the display of transcripts alongside translations can be beneficial to the users (Sperber et al., 2020). This paper addresses multilingual ST and investigates more closely the interactions between speech transcription (ASR) and speech translation (ST) in a multilingual end-to-end architecture based on Trans- former. While those interactions were previously investigated as a simple multi-task framework for a bilingual case (Anastasopoulos and Chiang, 2018), we propose a dual-decoder with an ASR decoder tightly coupled with an ST decoder and evaluate its effectiveness on one-to-many ST. Our …

Content truncated to 500 words