Weakly-Supervised Modeling of Contextualized Event Embedding for Discourse Relations

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 489–498 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 489 ConceptBert: Concept-Aware Representation for Visual Question Answering Franc¸ois Gard`eres∗ Ecole Poyltechnique Paris, France Maryam Ziaeefard† McGill University Montreal, Canada Baptiste Abeloos Thales Montreal, Canada Freddy Lecue Inria, France Thales, Canada Abstract Visual Question Answering (VQA) is a chal- lenging task that has received increasing at- tention from both the computer vision and the natural language processing communities. Current works in VQA focus on questions which are answerable by direct analysis of the question and image alone. We present a concept-aware algorithm, ConceptBert, for questions which require common sense, or basic factual knowledge from external struc- tured content. Given an image and a ques- tion in natural language, ConceptBert requires visual elements of the image and a Knowl- edge Graph (KG) to infer the correct an- swer. We introduce a multi-modal represen- tation which learns a joint Concept-Vision- Language embedding. We exploit ConceptNet KG for encoding the common sense knowl- edge and evaluate our methodology on the Out- side Knowledge-VQA (OK-VQA) and VQA datasets. Our code is available at https:// github.com/ZiaMaryam/ConceptBERT 1 Introduction Visual Question Answering (VQA) was firstly in- troduced to bridge the gap between natural lan- guage processing and image understanding appli- cations in the joint space of vision and language (Malinowski and Fritz, 2014). Most VQA benchmarks compute a question rep- resentation using word embedding techniques and Recurrent Neural Networks (RNNs), and a set of object descriptors comprising bounding box coordi- nates and image features vectors. Word and image representations are then fused and fed to a network to train a VQA model. However, these approaches are practical when no knowledge beyond the visual content is required. ∗This work was done when the author was an intern at Thales. Contact email: francois.garderes@student.ecp.fr †This work was done when the author worked at Thales. Incorporating the external knowledge introduces several advantages. External knowledge and sup- porting facts can improve the relational representa- tion between the objects detected in the image, or between entities in the question and objects in the image. It also provides information on how the an- swer can be derived from the question. Therefore, the complexity of the questions can be increased based on the supporting knowledge base. Organizing the world’s facts and storing them in a structured database, large scale Knowledge Bases (KB), have become important resources for representing the external knowledge. A typical KB consists of a collection of subject-predicate- object triplets also known as a fact. A KB in this form is often called a Knowledge Graph (KG) (Bollacker et al.) due to its graphical rep- resentation. The entities are nodes and the rela- tions are the directed edges that link the nodes. The triples specify that two entities are connected by a particular relation, e.g., (Shakespeare, writerOf, Hamlet). A VQA system that exploits KGs is an emerg- ing research topic, and is not well-studied. Recent research has started integrating knowledge-based methods …

Content truncated to 500 words