Towards Unifying Reference Expression Generation and Comprehension
30568Authors
nan
Abstract
AbstractReference Expression Generation (REG) and Comprehension (REC) are two highly correlated tasks. Modeling REG and REC simultaneously for utilizing the relation between them is a promising way to improve both. However, the problem of distinct inputs, as well as building connections between them in a single model, brings challenges to the design and training of the joint model. To address the problems, we propose a unified model for REG and REC, named UniRef. It unifies these two tasks with the carefully-designed Image-Region-Text Fusion layer (IRTF), which fuses the image, region and text via the image cross-attention and region cross-attention. Additionally, IRTF could generate pseudo input regions for the REC task to enable a uniform way for sharing the identical representation space across the REC and REG. We further propose Vision-conditioned Masked Language Modeling (VMLM) and Text-Conditioned Region Prediction (TRP) to pre-train UniRef model on multi-granular corpora. The VMLM and TRP are directly related to REG and REC, respectively, but could help each other. We conduct extensive experiments on three benchmark datasets, RefCOCO, RefCOCO+ and RefCOCOg. Experimental results show that our model outperforms previous state-of-the-art methods on both REG and REC.
Keywords
Full Content
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6598 - 6611 December 7-11, 2022 ©2022 Association for Computational Linguistics 6598 6599 Image X-Att FFN Self Trans-Dec Layer Language Encoder Vision Encoder ×"!" ×"!# IRTF Region X-Att Box Head LM Head people dog cat guy ride tree REC® Fusion (31,167,91,284) Region Predictor horse with no [MASK] horse with no ride REG: REC: Region feature Image feature Text feature Region feature REC REG 6600 6601 6602 6603 Image Cross-attention Region Cross-attention sheep with ear not visible Image Cross-attention Region Cross-attention chair with bird on it 6604 (m) plant (n) flowerpot (o) woman with glasses (p) woman next to the woman with glasses (q) person farthest from the woman with glasses (r) plate closest to the woman with glasses (a) the tallest (b) man without glasses (c) man not wearing blue jeans (d) first person from left to right (e) second person from left to right (f) third person from left to right (g) desk lamp (i) tv controller (h) table with lamp (j) table with tv controller (k) table between two chairs (l) chair farthest from tv 6605 6606 6607 6608 6609 top right bowl bottom right guy girl in pink right zebra bottom hot dog umbrella (a) RefCOCO red jacket Woman in black on right second from left catcher little girl in pink girl in mirror flower shirt sandwich half closest to us white and blue bus sitting down with white shorts woman in white giraffe closest to us book with people on it partial bus (b) RefCOCO+ a small green tree in the middle of a group of people a little girl in a white shirt the half of the sandwich on the right a red container with a slice of meat in it the cow on the far left a vase behind two other vases (c) RefCOCOg the polar bear on the left a brown wooden boat with white writing on the side a man in a blue jacket a black chicken laying down in the grass 6610 zebra on the left apple in front boy holding toy crib middle guy front light (a) RefCOCO big tie top doughnut fatty boom boom right truck yellow boots pizza closest to you fourth orange on lowest row empty glass kid standing blue vest talolest man laying down smaller fridge elephant behind baby (b) RefCOCO+ a girl wearing glasses and a pink shirt a parked white ford suv hotel bed man is laying on the woman in a white shirt and black pants the handle of some luggage a light brown teddy bear with a white stomach baseball player base man tagging a runner (c) RefCOCOg a man carrying a surfboard dog next to another that has more black on it s face the dessert that is is the person s left hand 6611
Labeled Text
AbstractReference Expression Generation (REG) and Comprehension (REC) are two highly correlated tasks. Modeling REG and REC simultaneously for utilizing the relation between them is a promising way to improve both. However, the problem of distinct inputs, as well as building connections between them in a single model, brings challenges to the design and training of the joint model. To address the problems, we propose a unified model for REG and REC, named UniRef. It unifies these two tasks with the carefully-designed Image-Region-Text Fusion layer (IRTF), which fuses the image, region and text via the image cross-attention and region cross-attention. Additionally, IRTF could generate pseudo input regions for the REC task to enable a uniform way for sharing the identical representation space across the REC and REG. We further propose Vision-conditioned Masked Language Modeling (VMLM) and Text-Conditioned Region Prediction (TRP) to pre-train UniRef model on multi-granular corpora. The VMLM and TRP are directly related to REG and REC, respectively, but could help each other. We conduct extensive experiments on three benchmark datasets, RefCOCO, RefCOCO+ and RefCOCOg. Experimental results show that our model outperforms previous state-of-the-art methods on both REG and REC.
Publication Details
- Paper ID:
- 30568
- Year:
- 2022
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan