CLaC-np at SemEval-2021 Task 8: Dependency DGCNN

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 37–50 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 37 SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning Boyuan Zheng2∗, Xiaoyu Yang1, Yu-Ping Ruan3, Zhenhua Ling3, Quan Liu3, Si Wei4, Xiaodan Zhu1 1Queen’s University, Kingston, Canada; 2Northeastern University, Shenyang, China 3University of Science and Technology of China; 4iFlytek Research, Hefei, China steven.zheng010@gmail.com; xiaoyu.yang@queensu.ca; {quanliu,zhling}@ustc.edu.cn; siwei@iFlytek.com.cn; xiaodan.zhu@queensu.ca Abstract This paper introduces the SemEval-2021 shared task 4: Reading Comprehension of Abstract Meaning (ReCAM). This shared task is designed to help evaluate the ability of machines in representing and understanding abstract concepts. Given a passage and the corresponding question, a participating system is expected to choose the correct answer from five candidates of abstract concepts in a cloze-style machine reading comprehension setup. Based on two typical definitions of abstractness, i.e., the imperceptibility and nonspecificity, our task provides three subtasks to evaluate the participating models. Specifically, Subtask 1 aims to evaluate how well a system can model concepts that cannot be directly perceived in the physical world. Subtask 2 focuses on models’ ability in com- prehending nonspecific concepts located high in a hypernym hierarchy given the context of a passage. Subtask 3 aims to provide some insights into models’ generalizability over the two types of abstractness. During the SemEval-2021 official evaluation period, we received 23 submissions to Subtask 1 and 28 to Subtask 2. The participating teams addi- tionally made 29 submissions to Subtask 3. The leaderboard and competition website can be found at https://competitions .codalab.org/competitions/26153. The data and baseline code are available at https://github.com/boyuanzheng010/ SemEval2021-Reading-Comprehension- of-Abstract-Meaning. 1 Introduction Humans use words with abstract meaning in their daily life. In the past, research efforts have been exerted to better understand and model abstract meaning (Turney et al., 2011; Theijssen et al., ∗ This work was performed when Boyuan Zheng visited Queen’s University. 2011; Changizi, 2008; Spreen and Schulz, 1966). Modelling abstract meaning is closely related to many other NLP tasks such as reading compre- hension, metaphor modelling, sentiment analysis, summarization, and word sense disambiguation. In the past decade, significant advancement has been seen in developing computational models for semantics, based on deep neural networks. In this shared task, we aim to help assess the capability of the state-of-the-art deep learning models on repre- senting and modelling abstract concepts in a spe- cific reading comprehension setup. We introduce SemEval-2021 Task 4, Reading Comprehension of Abstract Meaning (ReCAM). Specifically, we design this shared task by follow- ing the machine reading comprehension framework (Hermann et al., 2015; Onishi et al., 2016; Hill et al., 2016), in which computers are given a pas- sage Di as well as a human summary Si to compre- hend. If a model can digest the passage as humans do, we expect it to predict the abstract word used in the summary, if the abstract word is masked. Unlike the previous work that requires comput- ers to predict concrete concepts, e.g., named enti- ties, in our task we ask …

Content truncated to 500 words