SemEval-2020 Task 5: Counterfactual Recognition

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 322–335 Barcelona, Spain (Online), December 12, 2020. 322 SemEval-2020 Task 5: Counterfactual Recognition Xiaoyu Yang† Stephen Obadinma† Huasha Zhao§ Qiong Zhang§ Stan Matwin‡ Xiaodan Zhu† † Ingenuity Labs Research Institute & ECE, Queen’s University, Canada § Alibaba Group, San Mateo, CA ‡ Department of Computer Science, Dalhousie University, Canada {xiaoyu.yang, 16sco, xiaodan.zhu}@queensu.ca qz.zhang@alibaba-inc.com creislerzhao@gmail.com stan@cs.dal.ca Abstract We present a counterfactual recognition (CR) task, the shared Task 5 of SemEval-2020. Coun- terfactuals describe potential outcomes (consequents) produced by actions or circumstances that did not happen or cannot happen and are counter to the facts (antecedent). Counterfac- tual thinking is an important characteristic of the human cognitive system; it connects an- tecedents and consequents with causal relations. Our task provides a benchmark for coun- terfactual recognition in natural language with two subtasks. Subtask-1 aims to determine whether a given sentence is a counterfactual statement or not. Subtask-2 requires the par- ticipating systems to extract the antecedent and consequent in a given counterfactual state- ment. During the SemEval-2020 official evaluation period, we received 27 submissions to Subtask-1 and 11 to Subtask-2. The data, baseline code, and leaderboard can be found at https://competitions.codalab.org/competitions/21691. The data and baseline code are also available at https://zenodo.org/record/3932442. 1 Introduction Counterfactual statements describe events that did not happen or cannot happen, and the possible conse- quences had those events happened, e.g., “if kangaroos had no tails, they would topple over” (Lewis, 2013). By developing a connection between the antecedent (e.g., “kangaroos had no tails”) and consequent (e.g., “they would topple over”), based on the imagination of possible worlds, humans can naturally form some causal judgments; e.g., having tails can prevent kangaroos from toppling over. One can understand counterfactuals using knowledge and explore the relationship between causes and effects. Although we may not be able to rollback the events which have happened or make impossible events occur in the real world, we can still think of potential outcomes of alternatives. Counterfactual thinking is a remarkable ability of human beings and is considered by many researchers, to act as the highest level of causation in the ladder of causal reasoning. Even the most advanced artificial intelligence system may still be far from achieving human-like counterfactual reasoning. Counterfactual reasoning is an important component for AI systems in obtaining stronger capability in generaliza- tion (Pearl and Mackenzie, 2018). Modeling counterfactuals has been studied in many different disciplines. For example, research in psychology has shown that counterfactual thinking can affect human cognition and behaviors (Epstude and Roese, 2008; Kray et al., 2010). The landmark paper of (Goodman, 1947) gives a detailed analysis of counterfactual conditionals in philosophy and logistics. As another example, counterfactuals have also been investigated in epidemiology to reveal the relationship between certain diseases and potential risk factors for those diseases (Vandenbroucke et al., 2016; Krieger and Davey Smith, 2016). We present a counterfactual recognition (CR) task, the task of determining whether a given statement conveys counterfactual thinking or not, and further …

Content truncated to 500 words