UoR at SemEval-2021 Task 4: Using Pre-trained BERT Token Embeddings for Question Answering of Abstract Meaning

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 799–804 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 799 UoR at SemEval-2021 Task 4: Using Pre-trained BERT Token Embeddings for Question Answering of Abstract Meaning Thanet Markchom, Huizhi Liang University of Reading White Knights, Berkshire, RG6 6AH United Kingdom t.markchom@pgr.reading.ac.uk, huizhi.liang@reading.ac.uk Abstract Most question answering tasks usually focus on predicting concrete answers, e.g., named entities. These tasks can be normally achieved by understanding the contexts without addi- tional information required. In Reading Com- prehension of Abstract Meaning (ReCAM) task, the abstract answers are introduced. To understand abstract meanings in the context, additional knowledge is essential. In this pa- per, we propose an approach that leverages the pre-trained BERT Token embeddings as a prior knowledge resource. According to the re- sults, our approach using the pre-trained BERT outperformed the baselines. It shows that the pre-trained BERT token embeddings can be used as additional knowledge for understand- ing abstract meanings in question answering. 1 Introduction Question answering (QA) is one of the machine reading comprehension tasks. The goal is to find an answer of a given question based on a given context. In most QA tasks (Chen et al., 2016; Lai et al., 2017), the answers are commonly concrete words appearing in the contexts. Abstract words, on the other hand, have usually been ignored in such tasks. Unlike concrete words, these abstract words are difficult to understand since they cannot be perceived directly with human senses. To study machine comprehension of abstract meaning, a task called Reading Comprehension of Abstract Mean- ing (ReCAM) (Zheng et al., 2021) was proposed. Unlike other QA tasks, the answers in ReCAM are abstract words that used to summarise the informa- tion in the contexts. ReCAM is divided into three subtasks. For sub- task 1, the abstract answers are in the form of imper- ceptible words such as ‘objective’, ‘chance’, and ‘prospective’. Subtask 2 is about nonspecificity. The answers in this subtask are abstract words that represent nonspecific or general concepts such as ‘vertebrate’. Subtask 3 focuses on generality of the models developed in the first two subtasks. In this subtask, the model in subtask 1 must be evaluated on subtask 2 data and vice versa. Based on these abstract answers, a machine has to understand the abstract meaning of certain words. This requires an additional knowledge to fulfill the lack of abstract concept information (Bi et al., 2019). For example, given a context which is a passage and a question shown in Table 1. In the example of Table 1, the context contains a passage and a question. The answer ‘neglected’ does not appear anywhere in the given passage. However, the sentence, ‘it grad- ually fell into disrepair in the late 1980s’, provides a clue to answer the question. If the model has a prior knowledge that the word ‘disrepair’ have a connection with ‘neglect’, then the correct answer will be chosen. In this work, we propose to use a pre-trained …

Content truncated to 500 words