Multimodal Robustness for Neural Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8505 - 8516 December 7-11, 2022 ©2022 Association for Computational Linguistics Multimodal Robustness for Neural Machine Translation Yuting Zhao∗ Tokyo Metropolitan University zhao-yuting@ed.tmu.ac.jp Ioan Calapodescu Naver Labs Europe ioan.calapodescu@naverlabs.com Abstract In this paper, we look at the case of a Generic text-to-text NMT model that has to deal with data coming from various modalities, like speech, images, or noisy text extracted from the web. We propose a two-step method, based on composable adapters, to deal with this problem of Multi- modal Robustness. In the first step, we sep- arately learn domain adapters and modality specific adapters, to deal with noisy input coming from various sources: ASR, OCR, or noisy text (UGC). In a second step, we com- bine these components at runtime via dy- namic routing or, when the source of noise is unknown, via two new transfer learning mechanisms (Fast Fusion and Multi Fusion). We show that our method provides a flex- ible, state-of-the-art, architecture able to deal with noisy multimodal inputs. 1 Introduction Neural Machine Translation (NMT) has achieved great performances (Gehring et al., 2017; Vaswani et al., 2017) but still suffers from various robustness problems, as shown by many previous works (Koehn and Knowles, 2017; Be- linkov and Bisk, 2017; Khayrallah and Koehn, 2018). Specific datasets like Michel and Neubig (2018); Berard et al. (2019a); Li et al. (2019a); Specia et al. (2020a), consisting in noisy com- ments from Reddit, or from Wikipedia or from restaurant reviews, were proposed to showcase this robustness problem. Based on these datasets, several solutions were proposed to deal with the problem. For example, using synthetic noise, with data aug- mentation Vaibhav et al. (2019); Karpukhin et al. (2019) or other adversarial training meth- ods (Ebrahimi et al., 2018; Cheng et al., 2018, ∗Work done during an internship at Naver Labs Europe. Figure 1: Setting for a text-to-text NMT model ro- bust to clean and noisy multimodal data (coming from speech, images or a web page) on a specific domain 2019, 2020). But these solutions, based on purely artificial noise do not guarantee the best results on real noise (Michel et al., 2019). Another set of solutions make use of real noisy data to fine-tune or adapt generic models, so that they become more robust to realistic noise distributions (Michel and Neubig, 2018; Murakami et al., 2019; Helcl et al., 2019; Alam and Anastasopoulos, 2020; Berard et al., 2020). But, as shown in Specia et al. (2020a), these noise specific methods do not generalise well on domains or noise distributions not seen at training time. In this work, we want to propose an extensi- ble robustness solution for NMT able to over- come some of these limitations. To have a realistic setting, we propose to build a model able to translate several clean domains and their respective noisy versions, coming from various sources or modalities. As shown in Figure 1, we work with clean in-domain data and noisy versions …

Content truncated to 500 words