Enhancing Transformer for End-to-end Speech-to-Text Translation

Enhancing Transformer for End-to-end Speech-to-Text Translation Mattia A. Di Gangi1,2 Matteo Negri1 1Fondazione Bruno Kessler via Sommarive, 18, Povo, TN, Italy surname@fbk.eu Roldano Cattoni1 Roberto Dess`ı2∗ 2Universit`a degli Studi di Trento CIMeC, DISI name.surname@unitn.it Marco Turchi1 Abstract Neural end-to-end architectures have been recently proposed for spoken language translation (SLT), following the state-of- the-art results obtained in machine transla- tion (MT) and speech recognition (ASR). Motivated by this contiguity, we propose an SLT adaptation of Transformer (the state-of-the-art architecture in MT), which exploits the integration of ASR solutions to cope with long input sequences featur- ing low information density. Long audio representations hinder the training of large models due to Transformer’s quadratic memory complexity. Moreover, for the sake of translation quality, handling such sequences requires capturing both short- and long-range dependencies between bi- dimensional features. Focusing on Trans- former’s encoder, our adaptation is based on: i) downsampling the input with con- volutional neural networks, which enables model training on non cutting-edge GPUs, ii) modeling the bidimensional nature of the audio spectrogram with 2D compo- nents, and iii) adding a distance penalty to the attention, which is able to bias it towards short-range dependencies. Our experiments show that our SLT-adapted Transformer outperforms the RNN-based baseline both in translation quality and training time, setting the state-of-the-art performance on six language directions. ∗Work done during a summer internship at the Machine Translation Research Unit at Fondazione Bruno Kessler. ∗c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. 1 Introduction Neural encoder-decoder models (Sutskever et al., 2014) with attention (Bahdanau et al., 2015) is a general architecture that, by enabling to tackle sequence-to-sequence problems with a single end- to-end model, achieved state-of-the-art results on machine translation (MT) (Bentivogli et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Chen et al., 2018) and obtained increasingly good per- formance in automatic speech recognition (Chan et al., 2016; Chiu et al., 2018; Zhang et al., 2017; Zeyer et al., 2018; Dong et al., 2018). The advan- tages of end-to-end techniques, besides their con- ceptual simplicity, reside on the prevention of er- ror propagation, and a reduced inference latency. Error propagation is particularly problematic for the SLT task (Ruiz et al., 2017), in which MT would be significantly penalized by errors result- ing from the previous ASR processing step. For this reason, end-to-end solutions have been re- cently proposed (B´erard et al., 2016; Weiss et al., 2017; Anastasopoulos and Chiang, 2018; Liu et al., 2018; Di Gangi et al., 2018) but, in terms of performance, they are still far behind the pipeline approach. The reason of the worse performance for this task can be found in its intrinsic diffi- culty, as it inherits and combines the challenges of the two pipelined tasks. Indeed, SLT mod- els map audio features into words, like in ASR, but the input is mapped into text in a different target language, like in MT. Thus, the problems of word reordering and …

Content truncated to 500 words