Adaptive Feature Selection for End-to-End Speech Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2533–2544 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2533 Adaptive Feature Selection for End-to-End Speech Translation Biao Zhang1 Ivan Titov1,2 Barry Haddow1 Rico Sennrich3,1 1School of Informatics, University of Edinburgh 2ILLC, University of Amsterdam 3Department of Computational Linguistics, University of Zurich B.Zhang@ed.ac.uk, {ititov,bhaddow}@inf.ed.ac.uk, sennrich@cl.uzh.ch Abstract Information in speech signals is not evenly distributed, making it an additional challenge for end-to-end (E2E) speech translation (ST) to learn to focus on informative features. In this paper, we propose adaptive feature se- lection (AFS) for encoder-decoder based E2E ST. We first pre-train an ASR encoder and apply AFS to dynamically estimate the im- portance of each encoded speech feature to ASR. A ST encoder, stacked on top of the ASR encoder, then receives the filtered fea- tures from the (frozen) ASR encoder. We take L0DROP (Zhang et al., 2020) as the backbone for AFS, and adapt it to sparsify speech fea- tures with respect to both temporal and fea- ture dimensions. Results on LibriSpeech En- Fr and MuST-C benchmarks show that AFS fa- cilitates learning of ST by pruning out ∼84% temporal features, yielding an average transla- tion gain of ∼1.3–1.6 BLEU and a decoding speedup of ∼1.4×. In particular, AFS reduces the performance gap compared to the cascade baseline, and outperforms it on LibriSpeech En-Fr with a BLEU score of 18.56 (without data augmentation).1 1 Introduction End-to-end (E2E) speech translation (ST), a paradigm that directly maps audio to a foreign text, has been gaining popularity recently (Duong et al., 2016; B´erard et al., 2016; Bansal et al., 2018; Di Gangi et al., 2019; Wang et al., 2019). Based on the attentional encoder-decoder framework (Bah- danau et al., 2015), it optimizes model parameters under direct translation supervision. This end-to- end paradigm avoids the problem of error propa- gation that is inherent in cascade models where an automatic speech recognition (ASR) model and 1We release our source code at https://github. com/bzhangGo/zero. play is not just child ’s games P L EY1 IH1 Z NAA1 T JH IH0 S T CH AY1 L D S G EY1 M Z −0.4 −0.2 0.0 0.2 0.4 Amplitude or Hz Figure 1: Example illustrating our motivation. We plot the amplitude and frequency spectrum of an audio segment (top), paired with its time-aligned words and phonemes (bot- tom). Information inside an audio stream is not uniformly distributed. We propose to dynamically capture speech fea- tures corresponding to informative signals (red rectangles) to improve ST. a machine translation (MT) model are chained to- gether. Nonetheless, previous work still reports that E2E ST delivers inferior performance compared to cascade methods (Niehues et al., 2019). We study one reason for the difficulty of train- ing E2E ST models, namely the uneven spread of information in the speech signal, as visualized in Figure 1, and the consequent difficulty of extract- ing informative features. Features corresponding to uninformative signals, such as pauses or noise, increase the input length and bring in …

Content truncated to 500 words