An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 823–839, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 823 Learning to Represent Image and Text with Denotation Graph Bowen Zhang ∗ † University of Southern California zhan734@usc.edu Hexiang Hu ∗ † University of Southern California hexiangh@usc.edu Vihan Jain Google Research vihanjain@google.com Eugene Ie Google Research eugeneie@google.com Fei Sha ‡ Google Research fsha@google.com Abstract Learning to fuse vision and language informa- tion and representing them is an important re- search problem with many applications. Re- cent progresses have leveraged the ideas of pre- training (from language modeling) and atten- tion layers in Transformers to learn representa- tion from datasets containing images aligned with linguistic expressions that describe the images. In this paper, we propose learning representations from a set of implied, visually grounded expressions between image and text, automatically mined from those datasets. In particular, we use denotation graphs to repre- sent how specific concepts (such as sentences describing images) can be linked to abstract and generic concepts (such as short phrases) that are also visually grounded. This type of generic-to-specific relations can be discov- ered using linguistic analysis tools. We pro- pose methods to incorporate such relations into learning representation. We show that state-of-the-art multimodal learning models can be further improved by leveraging auto- matically harvested structural relations. The representations lead to stronger empirical re- sults on downstream tasks of cross-modal im- age retrieval, referring expression, and com- positional attribute-object recognition. Both our codes and the extracted denotation graphs on the Flickr30K and the COCO datasets are publically available on https://sha-lab. github.io/DG. 1 Introduction There has been an abundant amount of aligned visual and language data such as text passages ∗Work done while at Google †Authors Contributed Equally ‡On leave from USC (feisha@usc.edu) describing images, narrated videos, subtitles in movies, etc. Thus, learning how to represent vi- sual and language information when they are se- mantically related has been a very actively studied topic. There are many vision + language applica- tions: image retrieval with descriptive sentences or captions (Barnard and Forsyth, 2001; Barnard et al., 2003; Hodosh et al., 2013; Young et al., 2014), im- age captioning (Chen et al., 2015; Xu et al., 2015), visual question answering (Antol et al., 2015), vi- sual navigation with language instructions (Ander- son et al., 2018b), visual objects localization via short text phrases (Plummer et al., 2015), and oth- ers. A recurring theme is to learn the representation of these two streams of information so that they cor- respond to each other, highlighting the notion that many language expressions are visually grounded. A standard approach is to embed the visual and the language information as points in a (joint) visual- semantic embedding space (Frome et al., 2013; Kiros et al., 2014; Faghri et al., 2018). One can then infer whether the visual information is aligned with the text information by checking how these points are distributed. How do we embed visual and text information? Earlier approaches focus on …

Content truncated to 500 words