HIT-SCIR at SemEval-2020 Task 5: Training Pre-trained Language Model with Pseudo-labeling Data for Counterfactuals Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 354–360 Barcelona, Spain (Online), December 12, 2020. 354 HIT-SCIR at SemEval-2020 Task 5: Training Pre-trained Language Model with Pseudo-labeling Data for Counterfactuals Detection Xiao Ding, Dingkui Hao, Yuewei Zhang, Kuo Liao, Zhongyang Li, Bing Qin, Ting Liu∗ Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China {xding, dkhao, ywzhang, kliao, zyli, qinb, tliu}@ir.hit.edu.cn Abstract We describe our system for Task 5 of SemEval 2020: Modelling Causal Reasoning in Language: Detecting Counterfactuals. Despite deep learning has achieved significant success in many fields, it still hardly drives today’s AI to strong AI, as it lacks of causation, which is a fundamental concept in human thinking and reasoning. In this task, we dedicate to detecting causation, especially counterfactuals from texts. We explore multiple pre-trained models to learn basic features and then fine-tune models with counterfactual data and pseudo-labeling data. Our team HIT-SCIR wins the first place (1st) in Sub-task 1 — Detecting Counterfactual Statements and is ranked 4th in Sub-task 2 — Detecting Antecedent and Consequence. In this paper we provide a detailed description of the approach, as well as the results obtained in this task 1. 1 Introduction Deep learning technologies have been truly remarkable and have caught many attentions of researchers. However, deep learning systems’ lack of understanding of causal relations is perhaps the biggest roadblock to giving them human-level intelligence (Pearl, 2019). Causation is a fundamental concept in human thinking and reasoning, which indicates a special semantic relation between the cause with the effect (Stukker et al., 2008). Pearl and Mackenzie (2018) propose that there are three levels of causation, and the top level is the counterfactual analysis. Counterfactual statements describe events that did not actually happen or cannot happen, as well as the possible consequence if the events have had happened (Yang et al., 2020). SemEval 2020 Task 5 consists of two subtasks which aim to detect counterfactual descriptions in sentences, this paper presents solutions to both of two subtasks. Subtask1 — Detecting counterfactual statements refers to determining whether a given statement is counterfactual or not in several domains. For example, given the sentence: “Her post-traumatic stress could have been avoided if a combination of paroxetine and exposure therapy had been prescribed two months earlier.”, the system is required to determine whether it is a counterfactual statement or not. This categorization task is evaluated by three evaluation indexes: Precision, Recall and F1. The challenges of this subtask include: a) positive and negative samples are unevenly distributed in the training data and test data. b) the patterns of counterfactual statements can be hardly defined by rules or learned by traditional word embedding based models. Subtask2 — Detecting antecedent and consequence aims to locate antecedent and consequent in counterfactuals. A counterfactual statement can be converted to a contrapositive with a true antecedent and consequent (Goodman and Nelson, 1947). Consider the “post-traumatic stress” example discussed above; it can be transposed into “because her post-traumatic stress was not avoided, …

Content truncated to 500 words