XRJL-HKUST at SemEval-2021 Task 4: WordNet-Enhanced Dual Multi-head Co-Attention for Reading Comprehension of Abstract Meaning

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 793–798 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 793 XRJL-HKUST at SemEval-2021 Task 4: WordNet-Enhanced Dual Multi-head Co-Attention for Reading Comprehension of Abstract Meaning Yuxin Jiang*, Ziyi Shou*, Qijun Wang, Hao Wu, Fangzhen Lin HKUST-Xiaoi Robot Joint Lab on Machine Learning and Cognitive Reasoning Department of Computer Science and Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong {yjiangcm,zshou,qwangcd,hwubx}@connect.ust.hk,flin@cse.ust.hk Abstract This paper presents our submitted system to SemEval 2021 Task 4: Reading Comprehen- sion of Abstract Meaning. Our system uses a large pre-trained language model as the en- coder and an additional dual multi-head co- attention layer to strengthen the relationship between passages and question-answer pairs, following the current state-of-the-art model DUMA. The main difference is that we stack the passage-question and question-passage at- tention modules instead of calculating paral- lelly to simulate re-considering process. We also add a layer normalization module to im- prove the performance of our model. Fur- thermore, to incorporate our known knowl- edge about abstract concepts, we retrieve the definitions of candidate answers from Word- Net and feed them to the model as extra in- puts. Our system, called WordNet-enhanced DUal Multi-head Co-Attention (WN-DUMA), achieves 86.67% and 89.99% accuracy on the official blind test set of subtask 1 and subtask 2 respectively. 1 Introduction Recently, there has been an increasing interest on Machine Reading Comprehension (MRC). While most MRC studies such as CNN/Daily Mail (Her- mann et al., 2015) focus on concrete concepts, Se- mEval 2021 Task 4, Reading Comprehension on Abstract Meaning (ReCAM), targets abstract con- cept understanding, including imperceptibility in subtask 1 and nonspecificity in subtask 2. The for- mer, imperceptibility, highlights the abstract words that refer to ideas and concepts that do not corre- spond directly to human perception. The latter is for hypernyms and abstract concepts such as the class of vertebrate which includes whales as a con- crete subclass (Changizi, 2008). *Equal contribution. Figure 1: An example of ReCAM subtask 1. In this task, given news fragments and incom- plete abstracts, the machine needs to select the most suitable abstract words from candidate an- swers. Figure 1 shows one example of ReCAM subtask 1. Passage is the news sections and Ques- tion is a human written summary in which abstract words have been removed. Machines are requested to choose abstract words from five candidates for replacing @placeholder. For this shared task, we regard both subtasks as multi-choice MRC tasks. Various deep neural networks and attention mechanisms (e.g. (Dhin- gra et al., 2017; Wang et al., 2018; Zhang et al., 2020a,b; Jin et al., 2020)) have been proposed to address these tasks. In our work, follow- ing the state-of-the-art model DUMA (Zhu et al., 2020), we adopt a Pre-trained Language Model (PrLM) as encoder and extend with an additional dual multi-head co-attention layer to strengthen the relationship between passages and question- answer pairs. For the dual multi-head attention layer, while DUMA …

Content truncated to 500 words