Phone Features Improve Speech Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2388–2397 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2388 Phone Features Improve Speech Translation Elizabeth Salesky7 and Alan W Blackì 7Johns Hopkins University ìCarnegie Mellon University esalesky@jhu.edu, awb@cs.cmu.edu Abstract End-to-end models for speech translation (ST) more tightly couple speech recognition (ASR) and machine translation (MT) than a traditional cascade of separate ASR and MT models, with simpler model architectures and the potential for reduced error propagation. Their perfor- mance is often assumed to be superior, though in many conditions this is not yet the case. We compare cascaded and end-to-end models across high, medium, and low-resource condi- tions, and show that cascades remain stronger baselines. Further, we introduce two methods to incorporate phone features into ST models. We show that these features improve both archi- tectures, closing the gap between end-to-end models and cascades, and outperforming previ- ous academic work – by up to 9 BLEU on our low-resource setting. 1 Introduction End-to-end models have become the common approach for speech translation (ST), but the performance gap be- tween these models and a cascade of separately trained speech recognition (ASR) and machine translation (MT) remains, particularly in low-resource conditions. Mod- els for low-resource ASR leverage phone1 information, but this information is not typically leveraged by cur- rent sequence-to-sequence ASR or speech translation models. We propose two methods to incorporate phone features into current neural speech translation models. We explore the existing performance gap between end- to-end and cascaded models, and show that incorporat- ing phone features not only closes this gap, but greatly improves the performance and training efficiency of both model architectures, particularly in lower-resource conditions. The sequences of speech features used as input for ST are ≈10 times longer than the equivalent sequence of characters in e.g. a text-based MT model. This impacts memory usage, the number of model parameters, and 1The term ‘phone’ refers to segments corresponding to a col- lection of fine-grained phonetic units, but which may separate allophonic variation: see Jurafsky and Martin (2000). training time. Multiple consecutive feature vectors can belong to the same phone, but the exact number depends on the phone and local context. Further, these speech features are continuously valued rather than discrete, such that a given phone will have many different instantiations across a corpus. Neural models learn to associate ranges of similarly valued feature vectors in a data-driven way, impacting performance in lower-resource conditions. Using phoneme-level in- formation provides explicit links about local and global similarities between speech features, allowing models to learn the task at hand more efficiently and yielding greater robustness to lower-resource conditions. We propose two simple heuristics to integrate phoneme-level information into neural speech transla- tion models: (1) as a more robust intermediate represen- tation in a cascade; and (2) as a concatenated embedding factor. We use the common Fisher Spanish–English dataset to compare with previous work, and simulate high-, mid-, and low-resource conditions to compare model performance across …

Content truncated to 500 words