Don’t Discard Fixed-Window Audio Segmentation in Speech-to-Text Translation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 203 - 219 December 7-8, 2022 ©2022 Association for Computational Linguistics Don’t Discard Fixed-Window Audio Segmentation in Speech-to-Text Translation Chantal Amrhein1 and Barry Haddow2 1Department of Computational Linguistics, University of Zurich 2School of Informatics, University of Edinburgh amrhein@cl.uzh.ch, bhaddow@ed.ac.uk Abstract For real-life applications, it is crucial that end- to-end spoken language translation models per- form well on continuous audio, without rely- ing on human-supplied segmentation. For on- line spoken language translation, where models need to start translating before the full utterance is spoken, most previous work has ignored the segmentation problem. In this paper, we com- pare various methods for improving models’ robustness towards segmentation errors and dif- ferent segmentation strategies in both offline and online settings and report results on transla- tion quality, flicker and delay. Our findings on five different language pairs show that a simple fixed-window audio segmentation can perform surprisingly well given the right conditions.1 1 Introduction End-to-end spoken language translation (SLT) has seen considerable advances in recent years. To ap- ply these findings to real online and offline SLT settings, we need to be able to process continu- ous audio input. However, most previous work on end-to-end SLT makes use of human-annotated, sentence-like gold segments both at training and test time which are not available in real-life set- tings. Unfortunately, SLT models that were trained on such gold segments often suffer a noticeable quality loss when applied to artificially split au- dio segments (Zhang et al., 2021; Tsiamas et al., 2022b). This also highlights that a good segmenta- tion is more important for SLT than for automatic speech recognition (ASR) because we need to split the audio into “translatable units”. For a cascade system, a segmenter/punctuator can be inserted between the ASR and machine translation (MT) model (Cho et al., 2017) in order to create suitable segments for the MT model. However for end-to- 1We publicly release our code and model outputs here: https://github.com/ZurichNLP/window_ audio_segmentation gold shas fixed merged Figure 1: Visualisation of the different audio segmenta- tion methods studied in this paper. end SLT systems, it is still not clear how to best translate continuous input. Solving this problem is very much an active re- search field that has mainly been tackled from two sides: (1) improving SLT models to be more robust towards segmentation errors (Gaido et al., 2020; Li et al., 2021; Zhang et al., 2021) and (2) developing strategies to split streaming audio into segments that resemble the training data more closely (Gaido et al., 2021; Tsiamas et al., 2022b). Both types of approaches were successfully used in recent years for the IWSLT offline SLT shared task (Ansari et al., 2020; Anastasopoulos et al., 2021, 2022) to trans- late audio without gold segmentations. However, they have not yet been tested systematically in the online SLT setup where translation starts before the full utterance is spoken. Recent editions of the IWSLT simultaneous speech translation shared task focused more on evaluation using the …

Content truncated to 500 words