Fusion of Detected Objects in Text for Visual Question Answering

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2131–2140, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2131 Fusion of Detected Objects in Text for Visual Question Answering Chris Alberti Jeffrey Ling∗ Michael Collins David Reitter Google Research {chrisalberti, jeffreyling, mjcollins, reitter}@google.com Abstract To advance models of multimodal context, we introduce a simple yet powerful neural ar- chitecture for data that combines vision and natural language. The “Bounding Boxes in Text Transformer” (B2T2) also leverages ref- erential information binding words to portions of the image in a single unified architecture. B2T2 is highly effective on the Visual Com- monsense Reasoning benchmark1, achieving a new state-of-the-art with a 25% relative reduc- tion in error rate compared to published base- lines and obtaining the best performance to date on the public leaderboard (as of May 22, 2019). A detailed ablation analysis shows that the early integration of the visual features into the text analysis is key to the effectiveness of the new architecture. A reference implementa- tion of our models is provided2. 1 Introduction It has long been understood that the meaning of a word is systematically and predictably linked to the context in which it occurs (e.g., Firth 1957; Harris 1954; Deerwester et al. 1990; Mikolov et al. 2013). Different notions of context have resulted in different levels of success with downstream NLP tasks. Recent neural architectures includ- ing Transformer (Vaswani et al., 2017) and BERT (Devlin et al., 2018) have dramatically increased our ability to include a broad window of potential lexical hints. However, the same capacity allows for multimodal context, which may help model the meaning of words in general, and also sharpen its understanding of instances of words in context (e.g., Bruni et al. 2014). ∗Work done as part of the Google AI residency. 1https://visualcommonsense.com 2https://github.com/google-research/ language/tree/master/language/question_ answering/b2t2 Q: What was [1] doing before he sat in his living room? A1: He was reading [10]. A2: He was taking a shower. ✓ A3: [0] was sleeping until the noise [1] was making woke him up. A4: He was sleeping in his bedroom. R1: His clothes are disheveled and his face is glistening like he’s sweaty. R2: [0] does not look wet yet, but [0] looks like his hair is wet, and bathrobes are what you wear before or after a shower. R3: He is still wearing his bathrobe. ✓ R4: His hair appears wet and there is clothing hanging in front of him on a line as if to dry. Figure 1: An example from the VCR dataset. The tasks consists in picking an answer A1−4, and then picking a rationale R1−4. The data contains explicit pointers in the text to bounding boxes in the image. In this paper, we consider visual context in ad- dition to language and show that the right inte- gration of visual and linguistic information can yield improvements in visual question answering. The challenge we …

Content truncated to 500 words