ITNLP at SemEval-2021 Task 11: Boosting BERT with Sampling and Adversarial Training for Knowledge Extraction

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 51–58 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 51 TA-MAMC at SemEval-2021 Task 4: Task-adaptive Pretraining and Multi-head Attention for Abstract Meaning Reading Comprehension Jing Zhang, Yimeng Zhuang, Yinpei Su∗ Samsung Research China-Beijing (SRC-B) {jing97.zhang, ym.zhuang, yinpei.su}@samsung.com Abstract This paper describes our system used in the SemEval-2021 Task 4 Reading Comprehen- sion of Abstract Meaning, achieving 1st for subtask 1 and 2nd for subtask 2 on the leader- board. We propose an ensemble of ELECTRA- based models with task-adaptive pretraining and a multi-head attention multiple-choice classifier on top of the pre-trained model. The main contributions of our system are 1) reveal- ing the performance discrepancy of different transformer-based pretraining models on the downstream task, 2) presentation of an effi- cient method to generate large task-adaptive corpora for pretraining. We also investigated several pretraining strategies and contrastive learning objectives. Our system achieves a test accuracy of 95.11 and 94.89 on subtask 1 and subtask 2 respectively. 1 Introduction Machine reading comprehension (MRC) is one of the key tasks for measuring machines’ abil- ity of understanding human languages and rea- soning, it can be used broadly in real world ap- plications such as Q&A systems and dialogue systems. MRC often comes in a triplet style {passage, question, answer}, given a context passage, questions related with this passage is asked, and the machine is expected to give the an- swers. The question-answer form can be question- answer pair, where the answer text is to be provided by machines, or statement form where the answer is to be filled in as cloze or multiple choices se- lection. By the type of answer formation, MRC can be divided into extractive and generative MRC, the former takes segments from the passage as the answer and the latter requires answer text gener- ation based on the understanding of the passage. ∗Contribution during Intership in Samsung Research China-Beijing. Generative MRC is harder than extractive MRC, since it requires more on information integration and reasoning besides focusing on relevant infor- mation. One of the classic MRC approach focuses on matching networks, various network structures have been proposed to capture the semantic inter- action within passages/questions/answers. Recent years, pre-trained language models (LMs) have brought non-trivial progress to the performance on MRC, and there’s a decline of complex matching networks (Zhang et al., 2020). Plugging matching networks on top of pre-trained LMs can see ei- ther improvements or degradation in performance (Zhang et al., 2020; Zhu et al., 2020). Multiple- choice MRC (MMRC) often lacks abundant train- ing data for deep neural networks (this might be caused by the expensive human labelling cost) and it results in a limitation to take full advantage of the pre-trained LMs. The SemEval-2021 task 4 Reading Comprehen- sion of Abstract Meaning (Zheng et al., 2021), is a multiple-choice English MRC task, aiming at investigating the machine’s ability to understand abstract concepts in two aspects: subtask 1, non- concrete concepts, e.g. …

Content truncated to 500 words