PINGAN Omini-Sinitic at SemEval-2021 Task 4:Reading Comprehension of Abstract Meaning

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 820–826 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 820 PINGAN Omini-Sinitic at SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning Ye Wang Yanmeng Wang Haijun Zhu Bo Zeng Zhenghong Hao Shaojun Wang Jing Xiao Ping An Technology, Beijing 100191, China {wangye430,wangyanmeng219,zhuhaijun416,zengbo345, haozhenghong145,wangshaojun851,xiaojing661}@pingan.com.cn Abstract This paper describes the winning system for subtask 2 and the second-placed system for subtask 1 in SemEval 2021 Task 4: Reading Comprehension of Abstract Meaning. We pro- pose to use pre-trained ELECTRA discrimina- tor to choose the best abstract word from five candidates. An upper attention and auto de- noising mechanism is introduced to process the long sequences. The experiment results demonstrate that this contribution greatly fa- cilitates the contextual language modeling in reading comprehension task. The ablation study is also conducted to show the validity of our proposed methods. 1 Introduction Reading Comprehension of Abstract Meaning (Re- CAM) (Zheng et al., 2021) is a cloze-style task, which takes a document and related human written abstract with one word replaced by a placeholder as input. The model is required to choose the best word from five candidates. The ReCAM consists of three subtasks. In subtask 1 and 2, participating systems are required to choose the best impercepti- ble concept and hyper-nyms concepts word respec- tively. Subtask 3 aims to evaluate performance of a system trained on one definition and test on the other. Traditional cloze-style reading comprehension model (SA reader) (Kadlec et al., 2016) uses at- tention to directly pick the answer from the con- text, which makes model incapable to answer the question where the answer does not appear in pas- sage. Furthermore, GA reader (Dhingra et al., 2017) adopts multi-hop attention mechanism to build query-specific representation of answer for ranking the candidates which is not part of passage. Pre-trained language models (Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019; Lan et al., 2020) have been widely adopted for context modeling in many Natural Language Processing tasks. These models are pre-trained on huge cor- pora with plain texts and can better model con- textual dependencies of tokens, thus enhance the performance of downstream approaches. As de- scribed in (Lai et al., 2017; Zhang et al., 2020; Chen et al., 2019; Zhu et al., 2018), they improved the performance of single-choice reading compre- hension tasks by introducing pre-trained model, but they takes excessive memory for concatenating each option with the question and the passage. GPT2 (Radford et al., 2018) has outperformed the SOTA result on cloze-style task CBT (Hill et al., 2016). As stated in (Radford et al., 2018), GPT2 computes the probability of each choice and the rest of the sentence conditioned on this choice ac- cording to the pre-trained model, the answer is the choice with highest probability. GPT2 outperforms RNN-based models without fine-tuning on CBT task, we assume that pre-trained language model has better potential to address the cloze-style prob- lem than fine-tuning the …

Content truncated to 500 words