IITK-RSA at SemEval-2020 Task 5: Detecting Counterfactuals

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 458–467 Barcelona, Spain (Online), December 12, 2020. 458 IITK-RSA at SemEval-2020 Task 5: Detecting Counterfactuals Anirudh Anil Ojha∗ Rohin Garg∗ Shashank Gupta∗ Ashutosh Modi Indian Institute of Technology Kanpur (IITK) {aaojha, sronin, gshasha}@iitk.ac.in ashutoshm@cse.iitk.ac.in Abstract This paper describes our efforts in tackling Task 5 (Yang et al., 2020) of SemEval-2020. The task involved detecting a class of textual expressions known as counterfactuals and separating them into their constituent elements. Counterfactual statements describe events that have not or could not have occurred and the possible implications of such events. While counterfactual reasoning is natural for humans, understanding these expressions is difficult for artificial agents due to a variety of linguistic subtleties. Our final submitted approaches were an ensemble of various fine-tuned transformer-based and CNN-based models for the first subtask and a transformer model with dependency tree information for the second subtask. We ranked 4th and 9th in the overall leaderboard. We also explored various other approaches that involved the use of classical methods, other neural architectures and the incorporation of different linguistic features. 1 Introduction Counterfactuals (Starr, 2019) are statements that describe events that did not (or cannot) happen, which implies that their truth cannot be empirically verified. They may be divided into two major categories (Iatridou, 2000). In the first category, the statement discusses possible outcomes had the event in question actually happened, and thus is called a counterfactual conditional. Specifically, it contains a logical segment equivalent to “if A, then B” where A did not occur. The “if A” part is called the antecedent and the “then B” is called the consequent. An example of this is “If we had changed the battery on time, the car wouldn’t have broken down.” A statement of the second type is called a counterfactual wish. They also describe alternate realities, but do not talk about outcomes and do not display a conditional structure. They are called so because they often display ‘wish’ vocabulary e.g. “I wish there was an easy way to categorise counterfactuals!” Here, the segment referring to the fictional event is called the antecedent, and there is no consequent. While there are other ways to categorise these logical forms, we believe that this system covers most of the examples seen in the dataset and provides clear intuition about their general structure. Readers interested in a more rigorous treatment may refer to (Lewis, 2013). The ability to reason via counterfactuals is considered a higher form of intellect, as one must extrapolate the truth using causal relationships, a commonsense understanding of the world and other related events. Hence, for artificially intelligent systems, counterfactual reasoning is critical for generalisation and explainability. In terms of more concrete applications, it has been hypothesised to be important for natural language understanding, goal determination (a major challenge in modern reinforcement learning) and error analysis (Ginsberg, 1986). Recently, Mothilal et al. (2020) have worked on systems that generate counterfactual explanations for why classifiers behave the way they do …

Content truncated to 500 words