Speech Translation and the End-to-End Promise: Taking Stock of Where We Are

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7409–7421 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7409 Speech Translation and the End-to-End Promise: Taking Stock of Where We Are Matthias Sperber Apple sperber@apple.com Matthias Paulik Apple mpaulik@apple.com Abstract Over its three decade history, speech transla- tion has experienced several shifts in its pri- mary research themes; moving from loosely coupled cascades of speech recognition and machine translation, to exploring questions of tight coupling, and finally to end-to-end models that have recently attracted much at- tention. This paper provides a brief survey of these developments, along with a discus- sion of the main challenges of traditional ap- proaches which stem from committing to inter- mediate representations from the speech recog- nizer, and from training cascaded models sep- arately towards different objectives. Recent end-to-end modeling techniques promise a principled way of overcoming these issues by allowing joint training of all model components and removing the need for explicit intermediate representations. However, a closer look reveals that many end-to-end models fall short of solving these issues, due to compromises made to address data scarcity. This paper provides a unifying categorization and nomenclature that covers both traditional and recent approaches and that may help researchers by highlighting both trade-offs and open research questions. 1 Introduction Speech translation (ST), the task of translating acoustic speech signals into text in a foreign lan- guage, is a complex and multi-faceted task that builds upon work in automatic speech recognition (ASR) and machine translation (MT). ST appli- cations are diverse and include travel assistants (Takezawa et al., 1998), simultaneous lecture trans- lation (F¨ugen, 2008), movie dubbing/subtitling (Sa- boo and Baumann, 2019; Matusov et al., 2019), lan- guage documentation and crisis response (Bansal et al., 2017), and developmental efforts (Black et al., 2002). Until recently, the only feasible approach has been the cascaded approach that applies an ASR to the speech inputs, and then passes the results on to an MT system. Progress in ST has come from two fronts: general improvements in ASR and MT models, and moving from the loosely-coupled cascade in its most basic form toward a tighter coupling. However, despite considerable efforts toward tight coupling, a large share of the progress has arguably been owed simply to general ASR and MT improvements.1 Recently, new modeling techniques and in partic- ular end-to-end trainable encoder-decoder models have fueled hope for addressing challenges of ST in a more principled manner. Despite these hopes, the empirical evidence indicates that the success of such efforts has so far been mixed (Weiss et al., 2017; Niehues et al., 2019). In this paper, we will attempt to uncover poten- tial reasons for this. We start by surveying models proposed throughout the three-decade history of ST. By contrasting the extreme points of loosely cou- pled cascades vs. purely end-to-end trained direct models, we identify foundational challenges: erro- neous early decisions, mismatch between spoken- style ASR outputs and written-style MT inputs, and loss of speech information …

Content truncated to 500 words