Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2873–2887 August 1–6, 2021. ©2021 Association for Computational Linguistics 2873 Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference? Luisa Bentivogli1, Mauro Cettolo1, Marco Gaido1,2, Alina Karakanta1,2, Alberto Martinelli2∗, Matteo Negri1, Marco Turchi1 1Fondazione Bruno Kessler 2University of Trento {bentivo,cettolo,mgaido,akarakanta,negri,turchi}@fbk.eu Abstract Five years after the first published proofs of concept, direct approaches to speech trans- lation (ST) are now competing with tradi- tional cascade solutions. In light of this steady progress, can we claim that the perfor- mance gap between the two is closed? Start- ing from this question, we present a system- atic comparison between state-of-the-art sys- tems representative of the two paradigms. Fo- cusing on three language directions (English– German/Italian/Spanish), we conduct auto- matic and manual evaluations, exploiting high- quality professional post-edits and annotations. Our multi-faceted analysis on one of the few publicly available ST benchmarks attests for the first time that: i) the gap between the two paradigms is now closed, and ii) the subtle dif- ferences observed in their behavior are not suf- ficient for humans neither to distinguish them nor to prefer one over the other. 1 Introduction Speech translation (ST) is the task of automatically translating a speech signal in a given language into a text in another language. Research on ST dates back to the late eighties and its evolution followed the development of the closely related fields of speech recognition (ASR) and machine translation (MT) that, since the very beginning, provided the main pillars for building the so-called cascade ar- chitectures. With the advent of deep learning, the neural networks widely used in ASR and MT have been adapted to develop a new direct ST paradigm. This approach aims to overcome known limitations of the cascade one (e.g. architectural complexity, error propagation) with a single encoder-decoder architecture that directly translates the source sig- nal bypassing intermediate representations. ∗∗ The work of Alberto Martinelli was carried out during an internship at Fondazione Bruno Kessler. Until now, the consolidated underlying technolo- gies and the richness of available data have upheld the supremacy of cascade solutions in industrial applications. However, architectural simplicity, re- duced information loss and error propagation are the ace up the sleeve of the direct approach, which has rapidly gained popularity within the research community in spite of the critical bottleneck repre- sented by data paucity. Within a few years after the first proofs of con- cept (B´erard et al., 2016; Weiss et al., 2017), the performance gap between the two paradigms has gradually decreased. This trend is mirrored by the findings of the International Workshop on Spoken Language Translation (IWSLT),1 a yearly evalu- ation campaign where direct systems made their first appearance in 2018. On English-German, for instance, the BLEU difference between the best cascade and direct models dropped from 7.4 points in 2018 (Niehues et al., 2018) to 1.6 points in 2019 (Niehues et …

Content truncated to 500 words