AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2539–2545 August 1–6, 2021. ©2021 Association for Computational Linguistics 2539 AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation Wuwei Huang, Dexin Wang, Deyi Xiong ∗ College of Intelligence and Computing, Tianjin University, Tianjin, China {wwhuang, 2019218007, dyxiong}@tju.edu.cn Abstract In end-to-end speech translation, acoustic rep- resentations learned by the encoder are usually fixed and static, from the perspective of the de- coder, which is not desirable for dealing with the cross-modal and cross-lingual challenge in speech translation. In this paper, we show the benefits of varying acoustic states according to decoder hidden states and propose an adaptive speech-to-text translation model that is able to dynamically adapt acoustic states in the de- coder. We concatenate the acoustic state and target word embedding sequence and feed the concatenated sequence into subsequent blocks in the decoder. In order to model the deep inter- action between acoustic states and target hid- den states, a speech-text mixed attention sub- layer is introduced to replace the convention- al cross-attention network. Experiment results on two widely-used datasets show that the pro- posed method significantly outperforms state- of-the-art neural speech translation models. 1 Introduction Speech-to-text translation (ST) aims at translating the source language speech into the text of the tar- get language. Approaches to ST can be roughly divided into two categories: end-to-end ST and cas- caded ST. Early research on ST is primarily using a cascaded model that combines a speech recog- nition (ASR) module with a machine translation component, both usually trained independently on speech and parallel corpora (Ney, 1999; Matusov et al., 2005). In contrast, end-to-end ST, which di- rectly translates the speech of the source language into text of the target language (Berard et al., 2016), not only avoids error propagation in the ASR-MT pipeline, but also greatly reduces inference latency. However, despite these advantages, end-to-end ST is confronted with its own challenging problem- ∗Corresponding author s: performing cross-modal translation and cross- lingual conversion in one shot. On the one hand, compared with text-to-text translation, end-to-end ST has to deal with acoustic inputs which are typ- ically longer than their corresponding text input- s. This makes the cross-modal source-target de- pendencies more complicated. On the other hand, compared with monotonic ASR, end-to-end speech translation usually handles non-monotonic cross- lingual conversion. Generally, end-to-end ST uses the seq2seq encoder-decoder framework (Sutskever et al., 2014) as the backbone for training and inference, where the encoder computes hidden states layer by layer according to speech inputs. The decoder yields tar- get translations word by word by attending to the fixed-after-computing hidden states of the encoder. Since the hidden states are static in the encoder, information only flows one direction: from the en- coder to the decoder. Given the cross-modal and cross-lingual challenge in end-to-end ST, we ar- gue that more sophisticated interaction between the encoder and decoder would be desirable. In this paper, we propose an adaptive ST (AdaST) model that incorporates acoustic states into …

Content truncated to 500 words