BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9064–9079 July 9-14, 2023 ©2023 Association for Computational Linguistics BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric Mingda Chen, Paul-Ambroise Duquenne, Pierre Andrews, Justine Kao, Alexandre Mourachko, Holger Schwenk∗, Marta R. Costa-jussà∗ Meta AI {mingdachen,padqn,mortimer,jtk,}@meta.com {alexmourachko,schwenk,costajussa}@meta.com Abstract End-to-End speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. This means that generated speech has to be automatically transcribed, making the evaluation dependent on the availability and quality of automatic speech recognition (ASR) systems. In this paper, we propose a text-free evaluation metric for end-to-end S2ST, named BLASER, to avoid the dependency on ASR systems. BLASER leverages a multilingual multimodal encoder to directly encode the speech segments for source input, translation output and refer- ence into a shared embedding space and com- putes a score of the translation quality that can be used as a proxy to human evaluation. To evaluate our approach, we construct training and evaluation sets from more than 40k human annotations covering seven language directions. The best results of BLASER are achieved by training with supervision from human rating scores. We show that when evaluated at the sentence level, BLASER correlates significantly better with human judgment compared to ASR- dependent metrics including ASR-SENTBLEU in all translation directions and ASR-COMET in five of them. Our analysis shows combin- ing speech and text as inputs to BLASER does not increase the correlation with human scores, but best correlations are achieved when using speech, which motivates the goal of our re- search. Moreover, we show that using ASR for references is detrimental for text-based met- rics. 1 1 Introduction Speech-to-Speech translation seeks to translate speech segments from one language into another. ∗ Equal Research Leadership Contribution 1Code is available at https://github.com/faceb ookresearch/stopes Historically, it has been implemented and evalu- ated as a concatenation of three systems: auto- matic speech recognition (ASR), machine transla- tion (MT) and text-to-speech (TTS) (Lavie et al., 1997; Lazzari, 2006). In recent years, there has been increasing interest in end-to-end approaches (Jia et al., 2019; Lee et al., 2022a). While end-to- end S2ST is becoming popular, researchers still rely on text-based metrics to evaluate model perfor- mance by automatically transcribing the generated speech segments (Jia et al., 2019). These cascaded metrics rely on ASR systems, which for a given language may not have enough quality or may not even be available (Javed et al., 2022). They are also inappropriate for languages lacking standard- ized writing systems (Salesky et al., 2021a), like Hokkien or Algerian Arabic. In this work, we propose the text-free metric BLASER for S2ST evaluation, sidestepping the dependency on ASR systems. In particular, we use LASER encoders that support multiple languages and modalities including text (Heffernan et al., 2022) and speech (Duquenne et al., 2021). We use the LASER encoders to directly embed speech segments into vectors and compute a score estimat- ing the quality of generation. We then construct training and evaluation datasets from more than 40k human …

Content truncated to 500 words