CUNI Neural ASR with Phoneme-Level Intermediate Step for~Non-Native~SLT at IWSLT 2020

Proceedings of the 17th International Conference on Spoken Language Translation (IWSLT), pages 191–199 July 9-10, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 191 CUNI Neural ASR with Phoneme-Level Intermediate Step for Non-Native SLT at IWSLT 2020 Peter Pol´ak and Sangeet Sagar and Dominik Mach´aˇcek and Ondˇrej Bojar Charles University Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics {polak,sagar,machacek,bojar}@ufal.mff.cuni.cz Abstract In this paper, we present our submission to the Non-Native Speech Translation Task for IWSLT 2020. Our main contribution is a pro- posed speech recognition pipeline that con- sists of an acoustic model and a phoneme-to- grapheme model. As an intermediate repre- sentation, we utilize phonemes. We demon- strate that the proposed pipeline surpasses commercially used automatic speech recogni- tion (ASR) and submit it into the ASR track. We complement this ASR with off-the-shelf MT systems to take part also in the speech translation track. 1 Introduction This paper describes our submission to Non-Native Speech Translation Task in IWSLT 2020 (Ansari et al., 2020). We participate in two sub-tracks: offline speech recognition and offline speech trans- lation from English into Czech and German. We focus on the speech recognition, proposing a robust pipeline consisting of two components — an acoustic model recognizing phonemes, and a phoneme-to-grapheme translation model, see Fig- ure 1. We decided to use phonemes as the interme- diate representation between the acoustic and the translation model because we believe that conven- tional grapheme representation is too constrained with complicated rules of mapping speech to a tran- script. This issue becomes immense when dealing with dialects and non-native speakers. Both models used in our pipeline are end-to- end deep neural networks, Jasper (Li et al., 2019) for the acoustic model and Transformer (Vaswani et al., 2017) for the phoneme-to-grapheme transla- tion model. For punctuating, truecasing, segmenting and translation into Czech and German, we use off- the-shelf systems provided by ELITR project. (optional) Phoneme-LM Rescoring KenLM Acoustic�Model Jasper Phoneme-to-Grapheme Transformer MFCCs sound phonemes Transcript Figure 1: The architecture of proposed model. The paper is organized as follows: Section 2 reviews related work. In Sections 3 and 4 we de- scribe models for our speech recognition pipeline and their training. In Section 5, we describe the punctuator, truecasor and segmenter, and machine translation into Czech and German in Section 6. We summarize our submissions in Section 7 and conclude in Section 8. 2 Related Work This section reviews the related work. 2.1 Phonemes and Acoustic Models Phones and phonemes are well-established mod- elling units in ASR. They have been used since the beginning of the technology in 1950s (Juang and Rabiner, 2005), for an empirical comparison of different linguistic units for sound representation, see Riley and Ljolje (1992). An important work popularizing neural networks in ASR to phonemes is Waibel et al. (1989). This work proposes using a time-delayed neural network (TDNN) to model acoustic-phonetic features and the temporal relationship between them. The au- thors demonstrate that the proposed TDNN can learn shift-invariant internal abstraction of speech and …

Content truncated to 500 words