The HW-TSC’s Speech-to-Speech Translation System for IWSLT 2023

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 62–78 July 13-14, 2023 c⃝2023 Association for Computational Linguistics Evaluating Multilingual Speech Translation Under Realistic Conditions with Resegmentation and Terminology Elizabeth SaleskyJ Kareem DarwishA Mohamed Al-BadrashinyA Mona DiabM Jan NiehuesK JJohns Hopkins University AaiXplain MMeta AI KKarlsruhe Institute of Technology esalesky@jhu.edu Abstract We present the ACL 60/60 evaluation sets for multilingual translation of ACL 2022 technical presentations into 10 target languages. This dataset enables further research into multilin- gual speech translation under realistic record- ing conditions with unsegmented audio and domain-specific terminology, applying NLP tools to text and speech in the technical domain, and evaluating and improving model robustness to diverse speaker demographics. 1 Introduction The NLP and speech communities are rapidly ex- panding, which has motivated increased interest in multilingual scientific communication and accessi- bility. From the automatic captioning at NAACL 2019 provided by Microsoft to the current ACL 60-60 initiative1 for the 60th anniversary of ACL at 2022, it is clear that transcription and translation in the technical domain is needed, desired, and still a disproportionate challenge for current models compared to standard datasets in these spaces. Translating technical presentations presents chal- lenging conditions, from domain-specific terminol- ogy and adaptation, to recordings often captured with a laptop microphone and light background noise, diverse speaker demographics as well as unsegmented speech typically 10-60 minutes in duration. We have curated evaluation sets from presentations at ACL 2022 which have been pro- fessionally transcribed and translated with the sup- port of ACL and the 60-60 initiative. In this pa- per we describe the methodology to create this dataset, considerations and methods to evaluate speech translation models with it, and open chal- lenges we believe this dataset may support research towards. We release all data and intermediate steps to support further research in this space. 1https://www.2022.aclweb.org/dispecialinitiative Figure 1: Multilingual translation of ACL presentations. We present the ACL 60/60 evaluation sets to en- able greater development of tools by the field for the field. Specifically, we hope that this data en- ables further research into speech translation and other NLP applications in the technical domain with resegmentation and terminology, given a di- verse speaker set and realistic recording conditions, with the goal of increased accessibility and multi- linguality. Our dataset is publicly available through the ACL Anthology.2 2 Evaluation under realistic conditions To evaluate transcription and translation under real- istic conditions may require different metrics than with e.g. provided segmentation. Here we present the necessary metrics in order to discuss the dataset creation process. 2.1 Resegmentation While most offline speech translation models are trained with provided segmentation, in an applica- tion setting segmentation is unlikely to be provided. 2https://aclanthology.org/2023.iwslt-1.2 62 Most models are typically unable to maintain out- put quality given audio of typical talk lengths (10+ minutes), necessitating the use of automatic seg- mentation methods. In order to evaluate output with variable segmentation, resegmentation to a fixed reference is necessary. The standard tool within the field for many years …

Content truncated to 500 words