Visual Choice of Plausible Alternatives: An Evaluation of Image-based Commonsense Causal Reasoning

Visual Choice of Plausible Alternatives: An Evaluation of Image-based Commonsense Causal Reasoning Jinyoung Yeo Gyeongbok Lee∗ Gengyu Wang∗ Seungtaek Choi POSTECH Yonsei University Yonsei Unversity Yonsei Unversity Hyunsouk Cho Reinald Kim Amplayo Seung-won Hwang POSTECH Yonsei University Yonsei University {jinyeo, prory}@postech.edu {alias_n, posuer, hist0613, rktamplayo, seungwonh}@yonsei.ac.kr Abstract This paper proposes the task of Visual COPA (VCOPA). Given a premise image and two alternative images, the task is to identify the more plausible alternative with their commonsense causal context. The VCOPA task is designed as its desirable machine system needs a more detailed understanding of the image, commonsense knowledge, and complex causal reasoning than state-of-the-art AI tech- niques. For that, we generate an evaluation dataset containing 380 VCOPA questions and over 1K images with various topics, which is amenable to automatic evaluation, and present the performance of baseline reasoning approaches as initial benchmarks for future systems. Keywords: Commonsense knowledge, Causality, Reasoning, Evaluation, Image understanding 1. Introduction Commonsense causal reasoning is one of the fundamen- tal research problems in Knowledge Representation & Rea- soning (KR) domain. It aims at understanding the general causal dependency between common events or actions. Re- cent efforts for such understanding are focused on mea- suring the plausibility of one event statistically leading to another, and in particular, are competing on an evaluation set called Choice of Plausible Alternatives (COPA) (Roem- mele et al., 2011), which is to select the more plausible al- ternative as a cause (or effect) of the premise as: Example 1 Premise: A janitor is cleaning the floor. What is cause? Alternative 1: There is a broken cup on the floor. Alternative 2: There is a cup of coffee on the table. For the purpose of this reasoning, the state-of-the-art, called CausalNet (Luo et al., 2016), harvests causality scores of cause-effect term pairs, e.g., (‘broken’,‘clean’), by mining their causal patterns, e.g., “If...broken ..., then... clean...”, from an extremely large text corpus (10TB). As a result, it achieved a remarkable accuracy (70.2%) from COPA. In the real world, the desirable reasoning ability should be required not only in KR domain but also in Computer Vi- sion (CV) domain. Toward the optimal goal of human-level intelligence, CV researchers have actively studied on Vi- sual/Video Question Answering (VQA) (Antol et al., 2015; Ye et al., 2017; Zhao et al., 2017), which is to understand textual questions and images and give correct textual an- swers by machine. However, such QA tasks are at an early stage of “reasoning”, being limited to object-level reason- ing, e.g., “What is the man holding in his hand?”. Beyond object-level, the event-level visual reasoning is at intersection between the top of KR and CV capabilities, as the boundaries between the two domains are crumbling down due to the huge success of neural-based image-to- text or text-to-image converting techniques (Karpathy et al., ∗Authors in alphabetical order with equal contribution Premise (effect) Alternative 1 Alternative 2 inter object Figure 1: An example of VCOPA question. If a premise im- age is effect, plausible alternatives image …

Content truncated to 500 words