Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1882–1896 June 6–11, 2021. ©2021 Association for Computational Linguistics 1882 Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks Siddharth Dalmia Brian Yan Vikas Raunak Florian Metze Shinji Watanabe Language Technologies Institute, Carnegie Mellon University, USA {sdalmia,byan}@cs.cmu.edu Abstract End-to-end approaches for sequence tasks are becoming increasingly popular. Yet for com- plex sequence tasks, like speech translation, systems that cascade several models trained on sub-tasks have shown to be superior, sug- gesting that the compositionality of cascaded systems simplifies learning and enables so- phisticated search capabilities. In this work, we present an end-to-end framework that ex- ploits compositionality to learn searchable hid- den representations at intermediate stages of a sequence model using decomposed sub-tasks. These hidden intermediates can be improved using beam search to enhance the overall per- formance and can also incorporate external models at intermediate stages of the network to re-score or adapt towards out-of-domain data. One instance of the proposed framework is a Multi-Decoder model for speech translation that extracts the searchable hidden intermedi- ates from a speech recognition sub-task. The model demonstrates the aforementioned bene- fits and outperforms the previous state-of-the- art by around +6 and +3 BLEU on the two test sets of Fisher-CallHome and by around +3 and +4 BLEU on the English-German and English- French test sets of MuST-C.1 1 Introduction The principle of compositionality loosely states that a complex whole is composed of its parts and the rules by which those parts are combined (Lake and Baroni, 2018). This principle is present in engineer- ing, where task decomposition of a complex system is required to assess and optimize task allocations (Levis et al., 1994), and in natural language, where paragraph coherence and discourse analysis rely on decomposition into sentences (Johnson, 1992; Kuo, 1995) and sentence level semantics relies on decomposition into lexical units (Liu et al., 2020b). 1All code and models are released as part of the ESPnet toolkit: https://github.com/espnet/espnet. Similarly, many sequence-to-sequence tasks that convert one sequence into another (Sutskever et al., 2014) can be decomposed to simpler sequence sub- tasks in order to reduce the overall complexity. For example, speech translation systems, which seek to process speech in one language and output text in another language, can be naturally decom- posed into the transcription of source language au- dio through automatic speech recognition (ASR) and translation into the target language through ma- chine translation (MT). Such cascaded approaches have been widely used to build practical systems for a variety of sequence tasks like hybrid ASR (Hinton et al., 2012), phrase-based MT (Koehn et al., 2007), and cascaded ASR-MT systems for speech translation (ST) (Pham et al., 2019). End-to-end sequence models like encoder- decoder models (Bahdanau et al., 2015; Vaswani et al., 2017), are attractive in part due to their sim- plistic design and the reduced need for hand-crafted features. However, studies have shown mixed re- sults compared to cascaded models …

Content truncated to 500 words