Ethics Sheets for AI Tasks

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 57 - 69 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning Moxin Li1, Fuli Feng2∗, Hanwang Zhang3, Xiangnan He2, Fengbin Zhu1, Tat-Seng Chua1 1National University of Singapore, 2University of Science and Technology of China 3Nanyang Technological University, limoxin@u.nus.edu, fulifeng93@gmail.com, hanwangzhang@ntu.edu.sg, xiangnanhe@gmail.com, zhfengbin@gmail.com, dcscts@nus.edu.sg Abstract Neural discrete reasoning (NDR) has shown remarkable progress in combining deep mod- els with discrete reasoning. However, we find that existing NDR solution suffers from large performance drop on hypothetical questions, e.g., “what the annualized rate of return would be if the revenue in 2020 was doubled”. The key to hypothetical question answering (HQA) is counterfactual thinking, which is a natural ability of human reasoning but difficult for deep models. In this work, we devise a Learn- ing to Imagine (L2I) module, which can be seamlessly incorporated into NDR models to perform the imagination of unseen counterfac- tual. In particular, we formulate counterfactual thinking into two steps: 1) identifying the fact to intervene, and 2) deriving the counterfactual from the fact and assumption, which are de- signed as neural networks. Based on TAT-QA, we construct a very challenging HQA dataset with 8,283 hypothetical questions. We apply the proposed L2I to TAGOP, the state-of-the- art solution on TAT-QA, validating the ratio- nality and effectiveness of our approach. 1 Introduction Neural discrete reasoning (Dua et al., 2019) is an emerging technique for machine reading compre- hension (Rajpurkar et al., 2016) which aims at answering numerical questions from textual (Dua et al., 2019) or hybrid (Zhu et al., 2021) context1. NDR combines deep neural network with discrete and symbolic reasoning (e.g., addition, sorting, or counting) (Dua et al., 2019) and enables the com- prehension of complex contexts and compositional questions, which is critical for many practical ap- plications such as automatic diagnosis (Wei et al., 2018) and robo-advisor (Fisch et al., 2019). Exist- ing state-of-the-art NDR models implement the nu- ∗Corresponding author. 1where hybrid includes textual and tabular data in this work merical reasoning process as neural network mod- ules (Ran et al., 2019; Herzig et al., 2020; Zhu et al., 2021), e.g., a graph neural network for sorting (Ran et al., 2019; Chen et al., 2020a). In this work, we extend NDR to hypothetical question answering (HQA), where the question consists of an assumption beyond the context (Fig- ure 1). The ability of HQA will undoubtedly en- hance the practical use of NDR due to the uni- versality of hypothetical questions. However, cur- rent NDR models face severe generalization failure on hypothetical questions. An empirical evidence on such vulnerability is that the state-of-the-art model (Zhu et al., 2021) encounters a sharp perfor- mance drop (F1 score drops from 68.6% to 3.8%) on the TAT-QA dataset when changing the ques- tions to be hypothetical by adding a related as- sumption (see details in Section 2, Table 3). We postulate that the failure …

Content truncated to 500 words