Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5980–5994 July 9-14, 2023 ©2023 Association for Computational Linguistics Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination Hao Fei1, Qian Liu2, Meishan Zhang3∗, Min Zhang3, Tat-Seng Chua1 1 Sea-NExT Joint Lab, School of Computing, National University of Singapore 2 Nanyang Technological University 3 Harbin Institute of Technology (Shenzhen) {haofei37, dcscts}@nus.edu.sg liu.qian@ntu.edu.sg mason.zms@gmail.com zhangmin2021@hit.edu.cn Abstract In this work, we investigate a more realis- tic unsupervised multimodal machine transla- tion (UMMT) setup, inference-time image-free UMMT, where the model is trained with source- text image pairs, and tested with only source- text inputs. First, we represent the input images and texts with the visual and language scene graphs (SG), where such fine-grained vision- language features ensure a holistic understand- ing of the semantics. To enable pure-text input during inference, we devise a visual scene hal- lucination mechanism that dynamically gener- ates pseudo visual SG from the given textual SG. Several SG-pivoting based learning objec- tives are introduced for unsupervised transla- tion training. On the benchmark Multi30K data, our SG-based method outperforms the best-performing baseline by significant BLEU scores on the task and setup, helping yield trans- lations with better completeness, relevance and fluency without relying on paired images. Fur- ther in-depth analyses reveal how our model advances in the task setting. 1 Introduction Current neural machine translation (NMT) has achieved great triumph (Sutskever et al., 2014; Bah- danau et al., 2015; Zhu et al., 2020), however in the cost of creating large-scale parallel sentences, which obstructs the development of NMT for the minor languages. Unsupervised NMT (UMT) has thus been proposed to relieve the reliance of paral- lel corpora (Artetxe et al., 2018; Chen et al., 2018). The core idea of UMT is to align the represen- tation spaces between two languages with alter- native pivot signals rather than parallel sentences, such as bilingual lexicons (Lample et al., 2018), multilingual language models (LM) (Conneau and Lample, 2019) and back-translation technique (Sen- nrich et al., 2016). Recent trends have considered ∗Corresponding author Avoid parallel sent. during training? Avoid paired img. during testing? • Supervised MMT General MMT Zhang et al. (2020) Fang and Feng (2022) Li et al. (2022) ✓ • Unsupervised MMT Chen et al. (2018) Su et al. (2019) Huang et al. (2020) ✓ This work ✓ ✓ Table 1: Practical unsupervised MMT requires the avoid- ance of not only parallel sentences during training, but also the paired image during inference (testing). the incorporation of visual information, i.e., mul- timodal machine translation (MMT) (Specia et al., 2016; Huang et al., 2016). Intuitively, visual modal- ity can serve as language-agnostic signals, pivoting different languages by grounding the same textual semantics into the common visual space. Therefore, solving UMT with visual contents as pivot becomes a promising solution, a.k.a., unsupervised MMT (UMMT) (Huang et al., 2020; Su et al., 2019). UMMT systems are trained with only the text- image pairs (<text-img>), …

Content truncated to 500 words