CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension

Findings of the Association for Computational Linguistics: EACL 2023, pages 2586–2596 May 2-6, 2023 ©2023 Association for Computational Linguistics CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension Zhi Zhang ILLC, University of Amsterdam z.zhang@uva.nl Helen Yannakoudakis Dept. of Informatics, King’s College London helen.yannakoudakis@kcl.ac.uk Xiantong Zhen United Imaging zhenxt@gmail.com Ekaterina Shutova ILLC, University of Amsterdam e.shutova@uva.nl Abstract The task of multimodal referring expression comprehension (REC), aiming at localizing an image region described by a natural language expression, has recently received increasing at- tention within the research comminity. In this paper, we specifically focus on referring expres- sion comprehension with commonsense knowl- edge (KB-Ref), a task which typically requires reasoning beyond spatial, visual or semantic information. We propose a novel framework for Commonsense Knowledge Enhanced Trans- formers (CK-Transformer) which effectively integrates commonsense knowledge into the representations of objects in an image, facilitat- ing identification of the target objects referred to by the expressions. We conduct extensive experiments on several benchmarks for the task of KB-Ref. Our results show that the proposed CK-Transformer achieves a new state of the art, with an absolute improvement of 3.14% accuracy over the existing state of the art 1. 1 Introduction Referring expression comprehension (REC) aims at locating a target object/region in an image given a natural language expression as input. The nature of the task requires multi-modal reasoning and joint visual and language understanding. In the past few years, several REC tasks and datasets have been proposed, such as RefCOCO (Yu et al., 2016), RefCOCOg (Mao et al., 2016) and RefCOCO+ (Yu et al., 2016) (RefCOCOs). These ‘conventional’ REC tasks typically focus on identifying an object based on visual or spatial information of the object, such as its colour, shape, location, etc.; therefore primarily evaluating a model’s reasoning abilities over visual attributes and spatial relationships. In practice, however, people often describe an ob- ject using non-visual or spatial information – con- sider, for example, the sentence (expression) “Give 1The code will be available in https://github.com/ FightingFighting/CK-Transformer me something soft but rich in starch to eat” (Wang et al., 2020). Such instances require reasoning be- yond spatial and visual attributes, and need to be in- terpreted with respect to the common sense knowl- edge (fact) embedded in the expressions, such as knowledge about which kind of objects are edible, soft and rich in starch in the given image. Recently, Wang et al. (2020) proposed a new dataset, KB-Ref, to evaluate the reasoning ability of a model over not only visual and spatial features but also com- monsense knowledge. The dataset is devised such that at least one piece of fact from a knowledge base (KB) is required for a target object (referred to by an expression) to be identified. Therefore, searching for appropriate facts from a KB is also crucial part in KB-Ref. In contrast to the only existing work (Wang et al., 2020), in which for each object candidate, the top-K facts with the highest cosine similarity between the av- eraged Word2Vec (Mikolov et al., …

Content truncated to 500 words