MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 169–179 July 13-14, 2023 c⃝2023 Association for Computational Linguistics MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation Dominik Macháˇcek1 and Ondˇrej Bojar1 and Raj Dabre2 Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics1 National Institute of Information and Communications Technology, Kyoto, Japan2 1{machacek,bojar}@ufal.mff.cuni.cz, 2raj.dabre@nict.go.jp Abstract There have been several meta-evaluation stud- ies on the correlation between human rat- ings and offline machine translation (MT) evaluation metrics such as BLEU, chrF2, BERTSCORE and COMET. These metrics have been used to evaluate simultaneous speech translation (SST) but their correlations with human ratings of SST, which has been recently collected as Continuous Ratings (CR), are un- clear. In this paper, we leverage the evaluations of candidate systems submitted to the English- German SST task at IWSLT 2022 and conduct an extensive correlation analysis of CR and the aforementioned metrics. Our study reveals that the offline metrics are well correlated with CR and can be reliably used for evaluating machine translation in simultaneous mode, with some limitations on the test set size. We conclude that given the current quality levels of SST, these metrics can be used as proxies for CR, al- leviating the need for large scale human evalua- tion. Additionally, we observe that correlations of the metrics with translation as a reference is significantly higher than with simultaneous in- terpreting, and thus we recommend the former for reliable evaluation. 1 Introduction The current approach to evaluate simultaneous speech translation (SST, Cho and Esipova, 2016; Ma et al., 2019) systems that have text as the output modality is to use automatic metrics which are de- signed for offline text-to-text machine translation (MT), alongside other measures for latency and sta- bility. Researchers tend to use offline metrics, such as BLEU (Papineni et al., 2002), chrF2 (Popovi´c, 2017), BERTSCORE (Zhang et al., 2020), COMET (Rei et al., 2020) and others (Freitag et al., 2022) in SST despite no explicit evidence that they correlate with human ratings. However, simultaneous speech-to-text transla- tion has different characteristics compared to of- fline text-to-text MT. For example, when the users are following subtitles in real-time, they have lim- ited time for reading and comprehension as they cannot fully control the reading pace by themselves. Therefore, they may be less sensitive to subtle grammar and factual flaws than while reading a text document without any time constraints. The human evaluation of SST should therefore reflect the simultaneity. The users may also prefer brevity and simplicity over verbatim word-for-word trans- lation. Even if the reference is brief and simpler than the original, there may be lots of variants that the BLEU score and other MT metrics may not evaluate as correct. Furthermore, SST and MT differ in their input modalities. MT sources are assumed to originate as texts, while the SST source is a speech given in a certain situation, accompanied by para-linguistic means and specific context knowledge shared by the speaker …

Content truncated to 500 words