BLCU-NLP at SemEval-2020 Task 5: Data Augmentation for Efficient Counterfactual Detecting

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 633–639 Barcelona, Spain (Online), December 12, 2020. 633 BLCU-NLP at SemEval-2020 Task 5: Data Augmentation for Efficient Counterfactual Detecting Chang Liu Beijing Language and Culture University liuchang2014@gmail.com Dong Yu Beijing Language and Culture University yudong blcu@126.com Abstract Counterfactuals describe events counter to facts and hence naturally involve common sense, knowledge, and reasoning. SemEval 2020 task 5 is focusing on this field. We participate in the subtask 1 and we use BERT as our system. Our Innovations are feature extraction and data augmentation. We extract and summarize features of counterfactual statements, augment counterfactual examples in training set with the help of these features, and two general methods of data augmentation is experimented in our work. We demonstrate the effectiveness of our approaches, which achieves 0.95 of subtask 1 in F1 while using only a subset of giving training set to fine-tune the BERT model, and our official submission achieves F1 0.802, which ranks us 16th in the competition. 1 Introduction Counterfactual statements describe events that did not actually happen or cannot happen, as well as the possible consequence if the events have had happened. Detecting counterfactual statements involves common sense, knowledge, and reasoning, it is also the basis for all down-stream counterfactual related causal inference analysis in natural language.(Son et al., 2017) The problem of counterfactual have been studied in many domains, like the perspective of literally logi- cal relations between the antecedent and consequent of counterfactual forms and the outcomes(Goodman, 1947), and conducting counterfactual thought experiments for hypothetical tests on historical events, policies, or other aspects of a society and assess them by political scientists(Tetlock and Belkin, 1996). (Son et al., 2017) use a combination of a rule-based approach and a supervised classifier to capture counterfactual statements from Twitter, which is close to our problem. SemEval 2020 task 5 focus on the problem of detecting counterfactual statements, as (Yang et al., 2020) described in their work. Subtask 1 of task 5 is a two-classification problem, in which we need to find out whether a statement is counterfactual or not. We choose to apply feature extraction and data augmentation to tackle this problem, for the biggest challenge exists when doing this task is unbalanced data: although the training set has 13,000 examples in total, seems to be enough to fine-tune a BERT model, the number of both classes is highly unbalanced: there are only 1454 counterfactual examples, and 11546 non-counterfactual examples seriously dilute the proportion of counterfactual examples in the training set. By observing the training set, we conclude a set of feature words and rules to describe the occurrence of these words in counterfactual examples. In terms of data augmentation, we use two different methods, perform experiments on both augmented datasets to determine which method is more effective. We limit the number of non-counterfactual examples, which keeps the same as the number of counterfactual examples, in actual used training set to balance data. Using our best model setting, we get …

Content truncated to 500 words