Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12441–12455 July 9-14, 2023 ©2023 Association for Computational Linguistics Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks Yun Tang⋆, Anna Y. Sun⋆, Hirofumi Inaguma⋆, Xinyue Chen♢∗, Ning Dong⋆, Xutai Ma⋆, Paden D. Tomasello⋆, Juan Pino⋆ Meta AI⋆, Carnegie Mellon University♢ yuntang.email@gmail.com, hirofumii@meta.com Abstract Transducer and Attention based Encoder- Decoder (AED) are two widely used frame- works for speech-to-text tasks. They are de- signed for different purposes and each has its own benefits and drawbacks for speech-to- text tasks. In order to leverage strengths of both modeling methods, we propose a solu- tion by combining Transducer and Attention based Encoder-Decoder (TAED) for speech-to- text tasks. The new method leverages AED’s strength in non-monotonic sequence to se- quence learning while retaining Transducer’s streaming property. In the proposed framework, Transducer and AED share the same speech encoder. The predictor in Transducer is re- placed by the decoder in the AED model, and the outputs of the decoder are conditioned on the speech inputs instead of outputs from an unconditioned language model. The proposed solution ensures that the model is optimized by covering all possible read/write scenarios and creates a matched environment for stream- ing applications. We evaluate the proposed ap- proach on the MUST-C dataset and the findings demonstrate that TAED performs significantly better than Transducer for offline automatic speech recognition (ASR) and speech-to-text translation (ST) tasks. In the streaming case, TAED outperforms Transducer in the ASR task and one ST direction while comparable results are achieved in another translation direction. 1 1 Introduction Neural based end-to-end frameworks have achieved remarkable success in speech-to-text tasks, such as automatic speech recognition (ASR) and speech-to- text translation (ST) (Li, 2021). These frameworks include Attention based Encoder-Decoder model- ing (AED) (Bahdanau et al., 2014), connectionist ∗Xinyue Chen contributed to this work during her intern- ship at Meta. 1https://github.com/facebookresearch/fairseq/tree/main/ examples/transformer_transducer. temporal classification (CTC) (Graves et al., 2006) and Transducer (Graves, 2012) etc. They are de- signed with different purposes and have quite dif- ferent behaviors, even though all of them could be used to solve the mapping problem from a speech input sequence to a text output sequence. AED handles the sequence-to-sequence learning by allowing the decoder to attend to parts of the source sequence. It provides a powerful and gen- eral solution that is not bound to the input/output modalities, lengths, or sequence orders. Hence, it is widely used for ASR (Chorowski et al., 2015; Chan et al., 2015; Zhang et al., 2020; Gulati et al., 2020; Tang et al., 2021), and ST (Berard et al., 2016; Weiss et al., 2017; Li et al., 2021; Tang et al., 2022). CTC and its variant Transducer are designed to handle monotonic alignment between the speech input sequence and text output sequence. A hard alignment is generated between speech features and target text tokens during decoding, in which every output token is associated or synchronized with an input speech feature. …

Content truncated to 500 words