Phrase Grounding by Soft-Label Chain Conditional Random Field

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5112–5122, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5112 Phrase Grounding by Soft-Label Chain Conditional Random Field Jiacheng Liu Julia Hockenmaier University of Illinois at Urbana-Champaign, Urbana, IL, USA 61801 {jl25, juliahmr}@illinois.edu Abstract The phrase grounding task aims to ground each entity mention in a given caption of an image to a corresponding region in that im- age. Although there are clear dependencies between how different mentions of the same caption should be grounded, previous struc- tured prediction methods that aim to capture such dependencies need to resort to approxi- mate inference or non-differentiable losses. In this paper, we formulate phrase grounding as a sequence labeling task where we treat candi- date regions as potential labels, and use neu- ral chain Conditional Random Fields (CRFs) to model dependencies among regions for ad- jacent mentions. In contrast to standard se- quence labeling tasks, the phrase grounding task is defined such that there may be mul- tiple correct candidate regions. To address this multiplicity of gold labels, we define so- called Soft-Label Chain CRFs, and present an algorithm that enables convenient end-to- end training. Our method establishes a new state-of-the-art on phrase grounding on the Flickr30k Entities dataset. Analysis shows that our model benefits both from the entity dependencies captured by the CRF and from the soft-label training regime. Our code is available at github.com/liujch1998/ SoftLabelCCRF 1 Introduction Given an image and a corresponding caption, the phrase grounding task aims to ground each en- tity mentioned by a noun phrase in the caption to a region in the image. Phrase grounding has at- tracted much research interest due to its applica- tion in downstream tasks including image caption- ing (Karpathy et al., 2014; Fang et al., 2015; Don- ahue et al., 2017; Xu et al., 2015), image retrieval (Chen et al., 2017a; Radenovic et al., 2016), and visual question answering (Agrawal et al., 2017; Yu et al., 2017, 2018a). (a) Dependency between en- tities. The visual relation- ship between grounding re- gions for “cheerleaders” and “a girl” should agree with context “toss ... high up into the air”. (b) Gold label multiplicity. The green box is the an- notated gold grounding re- gion for entity phrase “Old man”, while the orange dash boxes are region proposals with IoU ≥ 0.5 with gold. Figure 1: Example image-caption pairs from Flickr30k Entities, illustrating entity dependencies and gold label multiplicity. Phrase grounding systems typically work by ranking a set of candidate regions (Chen et al., 2017b; Yu et al., 2018b). Region proposals are generated from the image by a vision backbone model, without conditioning on the caption. Fea- tures of the phrase to be grounded are extracted, and subsequently interact with features of candi- date regions, to determine phrase-region compat- ibility. Candidate regions are then ranked based on this compatibility metric, and the highest- scored candidate region is …

Content truncated to 500 words