Learning to Ask Conversational Questions by Optimizing Levenshtein Distance

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 514–524 August 1–6, 2021. ©2021 Association for Computational Linguistics 514 Learning Relation Alignment for Calibrated Cross-modal Retrieval Shuhuai Ren1, Junyang Lin3, Guangxiang Zhao1, Rui Men3, An Yang3, Jingren Zhou3, Xu Sun1,2∗, Hongxia Yang3 1MOE Key Lab of Computational Linguistics, School of EECS, Peking University 2Center for Data Science, Peking University 3Alibaba Group, China shuhuai ren@stu.pku.edu.cn, {zhaoguangxiang,xusun}@pku.edu.cn {junyang.ljy,menrui.mr,ya235025}@alibaba-inc.com {jingren.zhou,yang.yhx}@alibaba-inc.com Abstract Despite the achievements of large-scale mul- timodal pre-training approaches, cross-modal retrieval, e.g., image-text retrieval, remains a challenging task. To bridge the semantic gap between the two modalities, previous studies mainly focus on word-region alignment at the object level, lacking the matching between the linguistic relation among the words and the vi- sual relation among the regions. The neglect of such relation consistency impairs the con- textualized representation of image-text pairs and hinders the model performance and the interpretability. In this paper, we first pro- pose a novel metric, Intra-modal Self-attention Distance (ISD), to quantify the relation con- sistency by measuring the semantic distance between linguistic and visual relations. In re- sponse, we present Inter-modal Alignment on Intra-modal Self-attentions (IAIS), a regular- ized training method to optimize the ISD and calibrate intra-modal self-attentions from the two modalities mutually via inter-modal align- ment. The IAIS regularizer boosts the perfor- mance of prevailing models on Flickr30k and MS COCO datasets by a considerable margin, which demonstrates the superiority of our ap- proach.1 1 Introduction Cross-modal retrieval, including image-text re- trieval, video-text retrieval, etc., has long been an important downstream task in cross-modal rep- resentation learning. Image-Text Retrieval (ITR) aims at modeling the similarity of image-text pairs and recalling the most relevant one. It remains quite challenging due to the heterogeneity of the data and the semantic gap between two different modalities. To bridge this gap, neural networks are responsi- ble for learning global representations of images ∗Corresponding Author 1Our code is available at https://github.com/ lancopku/IAIS A guy with a red shirt walking. with A guy a red shirt walking. Previous alignment: object level Our alignment: relation level a red shirt with A guy a red shirt walking. a red shirt 0.12 0.36 0.52 a red shirt A guy (the rest regions) with, walking. (the rest tokens) 0.47 0.23 0.30 Intra-modal Self-attention Disagreement ☹ A bad case of intra-modal self-attention disagreement Self-attention Attention distribution Figure 1: The upper part shows a comparison of previ- ous object-level alignment and our relation-level align- ment. The symbol ↔ denotes alignment and ��� de- notes the self-attention stems from a query. The lower panel gives a bad case of inconsistent textual and vi- sual relations. The region of “a red shirt” pays consid- erable attention to the region of the dog, which does not benefit the matching and is inconsistent with the self-attention of the corresponding phrase. and texts in a joint semantic space and aligning the images and texts with the same semantics (Faghri et …

Content truncated to 500 words