Multimodal Quality Estimation for Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1233–1240 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1233 Multimodal Quality Estimation for Machine Translation Shu Okabe∗ 1 Fr´ed´eric Blain∗ 2 Lucia Specia1,2 1Department of Computing, Imperial College London, UK 2Department of Computer Science, University of Sheffield, UK shurokabe@gmail.com f.blain@sheffield.ac.uk l.specia@imperial.ac.uk Abstract We propose approaches to Quality Estima- tion (QE) for Machine Translation that ex- plore both text and visual modalities for Multi- modal QE. We compare various multimodal- ity integration and fusion strategies. For both sentence-level and document-level pre- dictions, we show that state-of-the-art neural and feature-based QE frameworks obtain bet- ter results when using the additional modality. 1 Introduction Quality Estimation (QE) for Machine Translation (MT) (Blatz et al., 2004; Specia et al., 2009) aims to predict the quality of a machine-translated text without using reference translations. It estimates a label (a category, such as ‘good’ or ‘bad’, or a numerical score) for a translation, given text in a source language and its machine translation in a target language (Specia et al., 2018b). QE can oper- ate at different linguistic levels, including sentence and document levels. Sentence-level QE estimates the translation quality of a whole sentence, while document-level QE predicts the translation qual- ity of an entire document, even though in practice in literature the documents have been limited to a small set of 3-5 sentences (Specia et al., 2018b). Existing work has only explored textual context. We posit that to judge (or estimate) the quality of a translated text, additional context is paramount. Sentences or short documents taken out of context may lack information on the correct translation of certain (esp. ambiguous) constructions. Inspired by recent work on multimodal machine learning (Baltrusaitis et al., 2019; Barrault et al., 2018), we propose to explore the visual modality in addition to the text modality for this task. Multimodality through vision offers interesting opportunities for real-life data since texts are in- ∗Two authors contributed equally. Source (EN) Danskin Women’s Bermuda Shorts MT (FR) Bermuda Danskin f´eminines court Table 1: Example of incorrectly machine-translated text: the word shorts is used to indicate short trousers, but gets translated in French as court, the adjective short. Here multimodality could help to detect the er- ror (extracted from the Amazon Reviews Dataset of McAuley et al., 2015). creasingly accompanied with visual elements such as images or videos, especially in social media but also in domains such as e-commerce. Multimodal- ity has not yet been applied to QE. Table 1 shows an example from our e-commerce dataset in which multimodality could help to improve QE. Here, the English noun shorts is translated by the adjective court (for the adjective short) in French, which is a possible translation out of context. However, as the corresponding product image shows, this prod- uct is an item of clothing, and thus the machine translation is incorrect. External information can hence help identify mismatches between transla- tions which are difficult to find within the …

Content truncated to 500 words