An Anchor-based Relative Position Embedding Method for Cross-Modal Tasks

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5401 - 5413 December 7-11, 2022 ©2022 Association for Computational Linguistics An Anchor-based Relative Position Embedding Method for Cross-Modal Tasks Ya Wang1,∗, Xingwu Sun1,2,∗, Fengzong Lian1, Zhanhui Kang1, Chengzhong Xu2 Machine Learning Platform Department, Tencent1 State Key Lab of IOTSC, Department of Computer Science, University of Macau2 connorywang@tencent.com, sammsun@tencent.com faxonlian@tencent.com, kegokang@tencent.com, czxu@um.edu.mo Abstract Position Embedding (PE) is essential for trans- former to capture the sequence ordering of in- put tokens. Despite its general effectiveness verified in Natural Language Processing (NLP) and Computer Vision (CV), its application in cross-modal tasks remains unexplored and suf- fers from two challenges: 1) the input text to- kens and image patches are not aligned; 2) the encoding space of each modality is different, making it unavailable for feature comparison. In this paper, we propose a unified position embedding method for these problems, called AnChor-basEd Relative Position Embedding (ACE-RPE), in which we first introduce an anchor locating mechanism to bridge the se- mantic gap and locate anchors from different modalities. Then we conduct the distance cal- culation of each text token and image patch by computing their shortest paths from the located anchors. Last, we embed the anchor-based distance to guide the computation of cross- attention. In this way, it calculates cross-modal relative position embeddings for cross-modal transformer. Benefiting from ACE-RPE, our method obtains new SOTA results on a wide range of benchmarks, such as Image-Text Re- trieval on MS-COCO and Flickr30K, Visual Entailment on SNLI-VE, Visual Reasoning on NLVR2 and Weakly-supervised Visual Ground- ing on RefCOCO+. 1 Introduction Transformer (Vaswani et al., 2017) has shown ex- cellent performance in Natural Language Process- ing (NLP), Computer Vision (CV) as well as cross- modal tasks, including natural language inference (Devlin et al., 2018), image classification (Wu et al., 2021), visual question answering (Wu et al., 2017) and visual entailment (Xie et al., 2019), etc. Never- theless, transformer module lacks the capability to capture the ordering information of the input tokens ∗ : equal contribution because of the limitation of its self-attention mech- anism. Therefore, incorporating explicit position representations is crucial to improve the perfor- mance of transformer-based models (Devlin et al., 2018; Dosovitskiy et al., 2020). Generally, there are two mainstream position en- coding methods in transformer-based NLP and CV models, i.e., absolute position embedding (APE) and relative position embedding (RPE). APE meth- ods (Vaswani et al., 2017; Devlin et al., 2018; Doso- vitskiy et al., 2020) encode absolute positions of the input tokens with either trainable (Devlin et al., 2018) or fixed embedding (Vaswani et al., 2017). These position embeddings are added with the to- ken embeddings, which are then passed to the self- attention layer to calculate the token relationship considering their positional information. It has been verified effective in a variety of NLP (Wang et al., 2020; Devlin et al., 2018) and CV (Wu et al., 2021) tasks. On the other hand, RPE methods (Chu et al., 2021; Shaw et al., …

Content truncated to 500 words