Is “moby dick” a Whale or a Bird? Named Entities and Terminology in Speech Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1707–1716 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1707 Is “moby dick” a Whale or a Bird? Named Entities and Terminology in Speech Translation Marco Gaido1,2, Susana Rodríguez3, Matteo Negri1, Luisa Bentivogli1 and Marco Turchi1 1Fondazione Bruno Kessler 2University of Trento 3Independent Researcher {mgaido,negri,bentivo,turchi}@fbk.eu Abstract Automatic translation systems are known to struggle with rare words. Among these, named entities (NEs) and domain-specific terms are crucial, since errors in their translation can lead to severe meaning distortions. Despite their importance, previous speech translation (ST) studies have neglected them, also due to the dearth of publicly available resources tai- lored to their specific evaluation. To fill this gap, we i) present the first systematic analysis of the behavior of state-of-the-art ST systems in translating NEs and terminology, and ii) re- lease NEuRoparl-ST, a novel benchmark built from European Parliament speeches annotated with NEs and terminology. Our experiments on the three language directions covered by our benchmark (en→es/fr/it) show that ST sys- tems correctly translate 75–80% of terms and 65–70% of NEs, with very low performance (37–40%) on person names. 1 Introduction The translation of rare words is one of the main challenges for neural machine translation (NMT) models (Sennrich et al., 2016; Koehn and Knowles, 2017). Among rare words, named entities (NEs) and terminology are particularly critical: not only are they important to understand the meaning of a sentence (Li et al., 2013), but they are also dif- ficult to handle due to the small number of valid translation options. While common words can be rendered in the target language with synonyms or paraphrases, NEs and terminology offer less ex- pressive freedom, which is typically limited to one valid option. Under these conditions, translation errors often result in blatant (meaningless, hilar- ious, or even offensive) errors, which jeopardize users’ trust in the translation system. One example is “moby dick” (in lower case, as in the typical output of a speech recognition system): Google Translate1 returns mazikó poulí (massive bird) for 1Accessed on the 27th April 2021. Greek, while the translation contains profanities for other languages like Hungarian or Italian. Previous works in NMT tried to mitigate the problem by: i) integrating knowledge graphs (KG) into the models (Lu et al., 2019; Moussallem et al., 2019; Zhao et al., 2020a,b; Ahmadnia et al., 2020), ii) exploiting dedicated modules for NE translation (Yan et al., 2019), or iii) adding NE tags to the source (Ugawa et al., 2018; Zhou et al., 2020). For terminology, the so far proposed dictionary-based approaches (Hokamp and Liu, 2017; Chatterjee et al., 2017; Hasler et al., 2018; Dinu et al., 2019; Song et al., 2020; Dougal and Lonsdale, 2020) share the idea of enriching the source sentence with terms’ translations found in a dictionary (e.g. “this is a term”→“this is a #term#término#”). The problem is even more challenging in auto- matic speech recognition (ASR) and speech-to-text translation (ST), where a lookup into KGs or dictio- …

Content truncated to 500 words