BUT-FIT at SemEval-2020 Task 5: Automatic Detection of Counterfactual Statements with Deep Pre-trained Language Representation Models

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 437–444 Barcelona, Spain (Online), December 12, 2020. 437 BUT-FIT at SemEval-2020 Task 5: Automatic detection of counterfactual statements with deep pre-trained language representation models Martin Fajcik, Josef Jon, Martin Docekal, Pavel Smrz Brno University of Technology, Faculty of Information Technology 612 66 Brno, Czech Republic {ifajcik,ijon,idocekal,smrz}@fit.vutbr.cz Abstract This paper describes BUT-FIT’s submission at SemEval-2020 Task 5: Modelling Causal Reasoning in Language: Detecting Counterfactuals. The challenge focused on detecting whether a given statement contains a counterfactual (Subtask 1) and extracting both antecedent and consequent parts of the counterfactual from the text (Subtask 2). We experimented with various state-of-the-art language representation models (LRMs). We found RoBERTa LRM to perform the best in both subtasks. We achieved the first place in both exact match and F1 for Subtask 2 and ranked second for Subtask 1. 1 Introduction One of the concerns of SemEval-2020 Task 5: Modelling Causal Reasoning in Language: Detecting Counterfactuals (Yang et al., 2020) is to research the extent to which current state-of-the-art systems can detect counterfactual statements. A counterfactual statement, as defined in this competition, is a conditional composed of two parts. The former part is the antecedent – a statement that is contradictory to known facts. The latter is the consequent – a statement that describes what would happen had the antecedent held1. To detect a counterfactual statement, the system often needs to posses a commonsense world knowledge to detect whether the antecedent contradicts with it. In addition, such a system must have an ability to reason over consequences that would arise had the antecedent would have been true. In some cases, the consequent might not be present at all, but instead a sequence resembling consequent, but with no consequential statement, might be present. Figure 1 shows a set of examples drawn from the data. If that was my daughter, I would have asked If I did something wrong. Nike's revenue last quarter alone climbed a solid 5%, to $8.432 billion, and would have increased 7% had it not been for foreign-currency exchange. Al Sharpton's storefront headquarters in Harlem, the gatherings were solemn, spirited and reflected the fraught nature of what would have been the Rev. Figure 1: Three examples from the training data containing counterfactual statements. Antecedents are highlighted with red bold, consequents with blue bold italic. The last example has no consequent. Counterfactuals were studied across a wide spectrum of domains. For instance, logicians and philoso- phers focus on logical rules between parts of counterfactual and its outcome (Goodman, 1947). Political scientists conducted counterfactual thought experiments for hypothetical tests on historical events, policies or other aspects of society (Tetlock and Belkin, 1996). However, there is only a small amount of work in computational linguistics studying this phenomenon. SemEval-2020 Task 5 aims at filling this gap in the field. The challenge consists of two subtasks: 1. Detecting counterfactual statements – classify whether the sentence has a counterfactual statement. 2. Detecting antecedent and consequence – extract boundaries of antecedent …

Content truncated to 500 words