On the Impact of Noises in Crowd-Sourced Data for Speech Translation

On the Impact of Noises in Crowd-Sourced Data for Speech Translation Siqi Ouyang1, Rong Ye2, Lei Li1 1University of California, Santa Barbara, CA, USA siqiouyang@ucsb.edu, leili@cs.ucsb.edu 2ByteDance AI Lab, Shanghai, China yerong@bytedance.com Abstract Training speech translation (ST) models re- quires large and high-quality datasets. MuST- C is one of the most widely used ST bench- mark datasets. It contains around 400 hours of speech-transcript-translation data for each of the eight translation directions. This dataset passes several quality-control filters during creation. However, we find that MuST-C still suffers from three major quality issues: audio- text misalignment, inaccurate translation, and unnecessary speaker’s name. What are the im- pacts of these data quality issues for model de- velopment and evaluation? In this paper, we propose an automatic method to fix or filter the above quality issues, using English-German (En-De) translation as an example. Our ex- periments show that ST models perform bet- ter on clean test sets, and the rank of proposed models remains consistent across different test sets. Besides, simply removing misaligned data points from the training set does not lead to a better ST model.1 1 Introduction Speech-to-text translation (ST) aims to translate a speech of a certain language into a text trans- lation of another language. Recent advances of end-to-end ST models have been largely boosted by the release of large high-quality parallel datasets (Kocabiyikoglu et al., 2018; Di Gangi et al., 2019; Wang et al., 2021). A clean test set is essential to evaluate the effectiveness of proposed models, and a sizeable well-aligned training set is important to train powerful ST models (Wang et al., 2020). Currently, the most widely-used ST benchmark dataset is MuST-C (Di Gangi et al., 2019). consists of around 400 hours of speech-transcript-It translation data from English into eight languages (German, Spanish, French, Italian, Dutch, Por- tuguese, Romanian, and Russian). MuST-C was 1The code is released at https://github.com/ owaski/MuST-C-clean built upon English TED Talks, which are often transcribed and translated by voluntary human an- notators. A bilingual sentence-level text corpus is firstly constructed based on sentence segmen- tation and Gargantua alignment tool (Braune and Fraser, 2010). Then, the transcription is aligned to the corresponding audio tracks using Gentle forced aligner2 built on Kaldi ASR toolkit (Povey et al., 2011). During alignment, entire talks are discarded if greater than 15% of words cannot be recognized, and sentences are removed if none of the words was aligned. Though MuST-C passed through several quality- control filters, this dataset is still not perfect. Through manual checking, we find three major quality issues in the dataset – inaccurate trans- lation, audio-text misalignment, and unnecessary speaker’s name. Along with the three issues iden- tified, more importantly, we are interested in the following questions: Do they affect the robustness of end-to-end speech translation models trained on this corpus? Can we trust the results from existing works using this data? In order to answer the above questions, we pro- pose an automatic method to filter or fix the afore- mentioned errors in both …

Content truncated to 500 words