ISCAS at SemEval-2020 Task 5: Pre-trained Transformers for Counterfactual Statement Modeling

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 658–663 Barcelona, Spain (Online), December 12, 2020. 658 ISCAS at SemEval-2020 Task 5: Pre-trained Transformers for Counterfactual Statement Modeling Yaojie Lu1,3, Annan Li1,3, Hongyu Lin1, Xianpei Han1,2, Le Sun1,2 1Chinese Information Processing Laboratory 2State Key Laboratory of Computer Science Institute of Software, Chinese Academy of Sciences, Beijing, China 3University of Chinese Academy of Sciences, Beijing, China {yaojie2017,liannan2019,hongyu2016,xianpei,sunle}@iscas.ac.cn Abstract ISCAS participated in two subtasks of SemEval 2020 Task 5: detecting counterfactual statements and detecting antecedent and consequence. This paper describes our system which is based on pre- trained transformers. For the first subtask, we train several transformer-based classifiers for detect- ing counterfactual statements. For the second subtask, we formulate antecedent and consequence extraction as a query-based question answering problem. The two subsystems both achieved third place in the evaluation. Our system is openly released at https://github.com/casnlu/ISCAS- SemEval2020Task5. 1 Introduction Counterfactual statements describe events that did not actually happen or cannot occur, as well as the possible consequence if the events have had happened. Counterfactual detecting aims to identify counterfactual statements in language and understand antecedents and consequents in these statements. For instance, the following sentence is a counterfactual statement, and the underlined term is the antecedent, while the italic term is the consequence: Her post-traumatic stress could have been avoided if a combination of paroxetine and exposure therapy had been prescribed two months earlier. Once understanding the statement, we can accumulate the causal knowledge for “post-traumatic stress”, i.e., “a combination of paroxetine and exposure may help cure post-traumatic stress”. To model counter- factual semantics and reason in natural language, SemEval 2020 Subtask 5 provides an English benchmark for two basic problems: detecting counterfactual statements and detecting antecedent and consequence (Yang et al., 2020). We build our evaluation systems that are built on pre-trained transformer-based neural network models, which have shown significant improvements over conventional methods in many NLP fields (Devlin et al., 2019; Liu et al., 2020; Lan et al., 2020). Specifically, in subtask 1, several transformer-based classifiers are designed to detect counterfactual statements. Besides, because counterfactual antecedent expressions are usually expressed using some obvious conditional assumption connectives, such as if and wish. We also equip transformers with additional convolutional neural network to capture the above strong local context information. For subtask 2, we formulate antecedent and consequence extraction as a query-based question answering problem. Specifically, to effectively model context information in counterfactual statements, we design two different kinds of input queries for antecedents/consequences and regard counterfactual statements as given paragraphs. The rest of this paper is organized as follows. Section 2 introduces the background of pre-trained transformers. Section 3 describes the overview of our system for two subtasks. In Section 4-5, we describe the detailed experiment setup and the overall system performance on the two subtasks. Finally, we conclude this paper in Section 6. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/. 659 Pre-trained Transformer Pre-trained Transformer … CNN Classifier Classifier (a) …

Content truncated to 500 words