End-to-End Speech Translation for Code Switched Speech

Findings of the Association for Computational Linguistics: ACL 2022, pages 1435 - 1448 May 22-27, 2022 c⃝2022 Association for Computational Linguistics End-to-End Speech Translation for Code Switched Speech Orion Weller1*, Matthias Sperber2, Telmo Pires2, Hendra Setiawan2, Christian Gollan2, Dominic Telaar2, Matthias Paulik2 1Johns Hopkins University 2Apple oweller@cs.jhu.edu,sperber@apple.com Abstract Code switching (CS) refers to the phenomenon of interchangeably using words and phrases from different languages. CS can pose sig- nificant accuracy challenges to NLP, due to the often monolingual nature of the underly- ing systems. In this work, we focus on CS in the context of English/Spanish conversations for the task of speech translation (ST), gener- ating and evaluating both transcript and trans- lation. To evaluate model performance on this task, we create a novel ST corpus derived from existing public data sets.1 We explore vari- ous ST architectures across two dimensions: cascaded (transcribe then translate) vs end-to- end (jointly transcribe and translate) and uni- directional (source → target) vs bidirectional (source ↔ target). We show that our ST ar- chitectures, and especially our bidirectional end-to-end architecture, perform well on CS speech, even when no CS training data is used. 1 Introduction Over half of the world’s population is estimated to be bilingual. 2 Those that know multiple languages are prone to code switch, i.e., to interchangeably use words and phrases from two (or more) lan- guages in situations such as casual dialog, while traveling abroad, or simply to use a word they find more fitting (Myers-Scotton and Ury, 1977; Here- dia and Altarriba, 2001). In CS, the base language is referred to as the matrix language while the con- tributing language is called the embedded language (Myers-Scotton, 1995), where speakers often use the matrix language the majority of the time. Code switched language is challenging to both automatic speech recognition (ASR) and machine translation (MT) - and therefore also to the com- posite task of speech translation (ST). While a rich 1We make instructions and extra data needed to construct our CS data set available at https://github.com/apple/ml-code- switched-speech-translation 2BBC: https://bbc.in/3jgwzZ2 * Work done as an intern. Transcript (CS) Translation (En) Acá te tiene como constantemente escribiendo papers y reviews no cierto Here they're like constantly writing papers and reviews right Audio Figure 1: An example instance of the joint speech recognition and translation task for code-switching (CS). Red indicates English words in the transcript and their corresponding words in the translation, whereas blue indicates Spanish words in the transcript and their corresponding translation. amount of prior works exist on CS in the context of ASR (Lyu et al., 2006; Ahmed and Tan, 2012; Vu et al., 2012; Johnson et al., 2017; Yue et al., 2019) and MT (Sinha and Thakur, 2005; Winata et al., 2021; Zhang et al., 2021; Yang et al., 2020), there is little prior work in the context of ST. The aforementioned challenges to ASR, MT and ST arise largely due to the lack of CS data as well as the often monolingual nature of ASR systems, …

Content truncated to 500 words