BYteam at SemEval-2020 Task 5: Detecting Counterfactual Statements with BERT and Ensembles

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 640–644 Barcelona, Spain (Online), December 12, 2020. 640 BYteam at SemEval-2020 Task 5: Detecting Counterfactual Statements with BERT and Ensembles Yang Bai, Xiaobing Zhou* School of Information Science and Engineering Yunnan University, Yunnan, P.R. China *Corresponding author:zhouxb@ynu.edu.com Abstract We participate in the classification tasks of SemEval-2020 Task: Subtask1: Detecting counterfac- tual statements of semeval-2020 task5(Detecting Counterfactuals). This paper examines different approaches and models towards detecting counterfactual statements classification. We choose the Bert model. However, the output of Bert is not a good summary of semantic information, so in order to obtain more abundant semantic information features, we modify the upper layer structure of Bert. Finally, our system achieves an accuracy of 88.90 % and F1 score of 86.30 % by hard voting, which ranks 6th on the final leader board of the in subtask 1 competition. 1 Introduction Counterfactual statements describe events that did not happen or cannot happen, as well as the possible consequence if the events have had happened. More specifically, counterfactuals describe events counter to facts and hence naturally involve common sense, knowledge, and reasoning. Tackling this problem is the basis for all down-stream counterfactual related causal inference analysis in natural language. In this work, we present our results for the SemEval 2020 Shared Task 5 (Yang et al., 2020): Modelling Causal Reasoning in Language: Detecting Counterfactuals. Subtask 1 focuses on the binary distinction if a sentence is counterfactual or not. Our submission for Subtask A ranks 6th. For Subtask 1, we experiment with different neural network-based models such as Recurrent Neural Network(RNN) (Zaremba et al., 2014), Attention (Vaswani et al., 2017) and Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018). For the imbalanced data set, we find that the BERT performs best on our development set. So, we combine the results by different BERT models. The rest of the paper is organized as follows. Section 2 briefly shows the related work. Section 3 elaborates on our approach. It shows the pre-processing step and architecture of our model. Section 4 describes the data set, hyper-parameters adopted research methodology and our results. Finally, Section 5 concludes our work. 2 Related Work Detecting the counterfactual statement is becoming more and more important. To build an effective classifier, one of the major problems is to find the appropriate features. Normally, two types of features are applied: surface features like n-grams and word representations trained by the neural network. Traditionally, the classifiers are trained on different types of surface features with approaches like SVM, Random Forest, and Logistic Regression. Using pre-trained word embeddings for feature extraction is effective in multiple NLP tasks.(Wang et al., 2018; Wang et al., 2016) Traditionally word embeddings are extracted from shallow neural networks trained on a large swathe of texts required to learn the contextual representations of words. For example, there are skip-grams (Mikolov et al., 2013) and GloVe (Pennington et al., 2014). However, these embeddings are learned from an aggregation of all possible …

Content truncated to 500 words