Cross-Modality Relevance for Reasoning on Language and Vision

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7642–7651 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7642 Cross-Modality Relevance for Reasoning on Language and Vision Chen Zheng, Quan Guo, Parisa Kordjamshidi Department of Computer Science and Engineering, Michigan State University East Lansing, MI 48824 {zhengc12,guoquan,kordjams}@msu.edu Abstract This work deals with the challenge of learn- ing and reasoning over language and vision data for the related downstream tasks such as visual question answering (VQA) and natu- ral language for visual reasoning (NLVR). We design a novel cross-modality relevance mod- ule that is used in an end-to-end framework to learn the relevance representation between components of various input modalities under the supervision of a target task, which is more generalizable to unobserved data compared to merely reshaping the original representation space. In addition to modeling the relevance between the textual entities and visual entities, we model the higher-order relevance between entity relations in the text and object relations in the image. Our proposed approach shows competitive performance on two different lan- guage and vision tasks using public bench- marks and improves the state-of-the-art pub- lished results. The learned alignments of in- put spaces and their relevance representations by NLVR task boost the training efficiency of VQA task. 1 Introduction Real-world problems often involve data from mul- tiple modalities and resources. Solving a prob- lem at hand usually requires the ability to rea- son about the components across all the involved modalities. Examples of such tasks are visual ques- tion answering (VQA) (Antol et al., 2015; Goyal et al., 2017) and natural language visual reason- ing (NLVR) (Suhr et al., 2017, 2018). One key to intelligence here is to identify the relations between the modalities, combine and reason over them for decision making. Deep learning is a prominent technique to learn representations of the data for decision making for various target tasks. It has achieved supreme performance based on large scale corpora (Devlin et al., 2019). However, it is a challenge to learn joint representations for cross- modality data because deep learning is data-hungry. There are many recent efforts to build such multi- modality datasets (Lin et al., 2014; Krishna et al., 2017; Johnson et al., 2017; Antol et al., 2015; Suhr et al., 2017; Goyal et al., 2017; Suhr et al., 2018). Researchers develop models by joining features, aligning representation spaces, and using Trans- formers (Li et al., 2019b; Tan and Bansal, 2019). However, generalizability is still an issue when op- erating on unobserved data. It is hard for deep learning models to capture high-order patterns of reasoning, which is essential for their generalizabil- ity. There are several challenging research directions for addressing learning representations for cross- modality data and enabling reasoning for target tasks. First is the alignment of the representation spaces for multiple modalities; second is designing architectures with the ability to capture high-order relations for generalizability of reasoning; third is using pre-trained modules to make the most use of minimal data. …

Content truncated to 500 words