Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1835–1841 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1835 Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation Elizabeth Salesky1, Matthias Sperber2, Alan W Black1 1Carnegie Mellon University, USA 2Karlsruhe Institute of Technology, Germany esalesky@cs.cmu.edu Abstract Previous work on end-to-end translation from speech has primarily used frame-level fea- tures as speech representations, which cre- ates longer, sparser sequences than text. We show that a na¨ıve method to create compressed phoneme-like speech representations is far more effective and efficient for translation than traditional frame-level speech features. Specif- ically, we generate phoneme labels for speech frames and average consecutive frames with the same label to create shorter, higher-level source sequences for translation. We see im- provements of up to 5 BLEU on both our high and low resource language pairs, with a re- duction in training time of 60%. Our improve- ments hold across multiple data sizes and two language pairs. 1 Introduction The way translation input is represented has been shown to impact performance as well as how much data the model requires to train (Sennrich et al., 2016; Salesky et al., 2018; Cherry et al., 2018). The current standard approach for text- based translation is to segment words into sub- word units as a preprocessing step (Sennrich et al., 2016). Clustering common character sequences increases frequency of units in data and improves generalization to new word forms and rare words. End-to-end speech-to-text models are showing competitive results (Weiss et al., 2017; Bansal et al., 2018a,b; B´erard et al., 2018; Anastasopou- los and Chiang, 2018), but so far have not com- pared different ways to represent speech input. Unlike text, where discrete trainable embeddings are typically used, speech models typically use continuous features extracted from sliding win- dows (frames), held fixed during training. Frame- level features yield significantly longer, more sparsely-represented sequences than their text equivalents, and so speech models stand to benefit from learning compressed input representations. Previous works have reduced sequence lengths to make training more tractable through fixed-length downsampling. However, phonemes are variable lengths. Other work has shown promising results using phonemic representations and unsupervised term discovery from variable length sequences in MT and other domains, but as discrete units (Wilkinson et al., 2016; Bansal et al., 2017; Adams et al., 2016; Kamper et al., 2016; Dalmia et al., 2018b; Chung and Glass, 2018). Inspired by these works, we explore higher-level continuous speech embeddings for end-to-end speech translation. Specifically, we use alignment methods to gen- erate phoneme labels, and average consecutive frames with the same label to create phoneme- like feature vectors from variable numbers of frames. We use the Fisher Spanish-English and low-resource Mboshi-French datasets. We com- pare performance on the full Fisher dataset to smaller subsets as in Bansal et al. (2018b). As it is not possible to train a high-performing recog- nizer on many lower-resource tasks, we use a high- resource model applied cross-lingually to create phoneme …

Content truncated to 500 words