Speechformer: Reducing Information Loss in Direct Speech Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1698–1706 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1698 Speechformer: Reducing Information Loss in Direct Speech Translation Sara Papi 1,2 , Marco Gaido1,2 , Matteo Negri1, Marco Turchi1 1Fondazione Bruno Kessler 2University of Trento {spapi,mgaido,negri,turchi}@fbk.eu Abstract Transformer-based models have gained in- creasing popularity achieving state-of-the-art performance in many research fields including speech translation. However, Transformer’s quadratic complexity with respect to the in- put sequence length prevents its adoption as is with audio signals, which are typically repre- sented by long sequences. Current solutions resort to an initial sub-optimal compression based on a fixed sampling of raw audio fea- tures. Therefore, potentially useful linguistic information is not accessible to higher-level layers in the architecture. To solve this is- sue, we propose Speechformer, an architecture that, thanks to a reduced memory usage in the attention layers, avoids the initial lossy com- pression and aggregates information only at a higher level according to more informed lin- guistic criteria. Experiments on three language pairs (en→de/es/nl) show the efficacy of our solution, with gains of up to 0.8 BLEU on the standard MuST-C corpus and of up to 4.0 BLEU in a low resource scenario. 1 Introduction Speech-to-text translation (ST) has been tradition- ally approached with cascade architectures consist- ing of a pipeline of two sub-components (Stentiford and Steer, 1988; Waibel et al., 1991): an automatic speech recognition (ASR), which transforms the audio input into a textual representation, and a ma- chine translation (MT) model, which projects the transcript into the target language. A more recent approach consists in directly translating speech into target text using a single model (Bérard et al., 2016; Weiss et al., 2017). This direct solution has inter- esting advantages (Sperber and Paulik, 2020): i) it can better exploit audio information (e.g. prosody) during the translation phase, ii) it has lower la- tency, and iii) it is not affected by error propagation. The authors contributed equally. Thanks to these advantages, the initially huge per- formance gap with cascade systems has gradually closed (Ansari et al., 2020), motivating research towards further improvements. Direct ST models are fed with features extracted from the audio with high frequency (usually every 10ms). This, on average, makes the resulting input sequence of vectors ∼10 times longer than the cor- responding text, leading to an intrinsically redun- dant (i.e. long and repetitive) representation. For this reason, it is not possible to process speech data with a vanilla Transformer encoder (Vaswani et al., 2017), whose self-attention layers have quadratic memory complexity with respect to the input length. State-of-the-art architectures tackle the problem by collapsing adjacent vectors in a fixed way, i.e. by mapping a predefined number of vectors (usually 4) into a single one, either using strided convolu- tional layers (Bérard et al., 2018; Di Gangi et al., 2019; Wang et al., 2020a) or by stacking them (Sak et al., 2015). As a positive side effect, these length reduction solutions lower input redundancy. As …

Content truncated to 500 words