IIE-NLP-Eyas at SemEval-2021 Task 4: Enhancing PLM for ReCAM with Special Tokens, Re-Ranking, Siamese Encoders and Back Translation

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 199–204 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 199 IIE-NLP-Eyas at SemEval-2021 Task 4: Enhancing PLM for ReCAM with Special Tokens, Re-Ranking, Siamese Encoders and Back Translation Yuqiang Xie Luxi Xing Wei Peng Yue Hu§ Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China {xieyuqiang,xingluxi,pengwei,huyue}@iie.ac.cn Abstract This paper introduces our systems for all three subtasks of SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning. To help our model better represent and understand ab- stract concepts in natural language, we well- design many simple and effective approaches adapted to the backbone model (RoBERTa). Specifically, we formalize the subtasks into the multiple-choice question answering format and add special tokens to abstract concepts, then, the final prediction of QA is considered as the result of subtasks. Additionally, we em- ploy many finetuning tricks to improve the per- formance. Experimental results show that our approach gains significant performance com- pared with the baseline systems. Our system¶ achieves eighth rank (87.51%) and tenth rank (89.64%) on the official blind test set of sub- task 1 and subtask 2 respectively. 1 Introduction The computer’s ability in understanding, represent- ing, and expressing abstract meaning is a funda- mental problem towards achieving true natural lan- guage understanding. SemEval-2021 Task 4: Read- ing Comprehension of Abstract Meaning (ReCAM) provides a well-formed benchmark that aims to study the machine’s ability in representing and un- derstanding abstract concepts (Zheng et al., 2021). The Reading Comprehension of Abstract Mean- ing (ReCAM) task is divided into three subtasks, including Imperceptibility, Nonspecificility, and Interaction. Please refer to the task description paper (Zheng et al., 2021) for more details. To address the above challenges in ReCAM, we first formalize all subtasks as a type of multiple-choice §Corresponding author. ¶Our Code is publicly available at https://github. com/indexfziq/IIE-NLP-Eyas-SemEval2021. Question Answering (QA) task like (Xing et al., 2020). Recently, the large Pre-trained Language Models (PLMs), such as GPT-2 (Radford et al., 2019), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), demonstrate their excellent ability in various natural language understanding tasks (Wang et al., 2018; Zellers et al., 2018, 2019). So, we employ the state-of-the-art PLM, RoBERTa, as our backbone model. Moreover, we design many simple and effective approaches to improve the per- formance of the backbone model, such as adding special tokens, sentence re-ranking, label smooth- ing and back translation. This paper describes approaches for all subtasks developed by the IIE-NLP-Eyas Team (Natural Language Processing group of Institute of Infor- mation Engineering of the Chinese Academy of Sciences). Our contributions are summarized as the followings: • We design many simple and effective ap- proaches to improve the performance of the PLMs on all three subtasks, such as special tokens, sentence re-ranking, siamese encoders and back translation and label smoothing; • Experiments demonstrate that our proposed methods achieve significant improvements compared with baselines and we obtain the 8th-place in …

Content truncated to 500 words