BiQuAD: Towards QA based on deeper text understanding

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 105–115 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 105 BiQuAD: Towards QA based on deeper text understanding Frank Grimm Philipp Cimiano CIT-EC, Universität Bielefeld {fgrimm, cimiano}@cit-ec.uni-bielefeld.de Abstract Recent question answering and machine read- ing benchmarks frequently reduce the task to one of pinpointing spans within a certain text passage that answers the given question. Typ- ically, these systems are not required to ac- tually understand the text on a deeper level that allows for more complex reasoning on the information contained. We introduce a new dataset called BiQuAD that requires deeper comprehension in order to answer questions in both extractive and deductive fashion. The dataset consist of 4, 190 closed-domain texts and a total of 99, 149 question-answer pairs. The texts are synthetically generated soccer match reports that verbalize the main events of each match. All texts are accompanied by a structured Datalog program that represents a (logical) model of its information. We show that state-of-the-art QA models do not perform well on the challenging long form contexts and reasoning requirements posed by the dataset. In particular, transformer based state-of-the- art models achieve F1-scores of only 39.0. We demonstrate how these synthetic datasets align structured knowledge with natural text and aid model introspection when approaching com- plex text understanding. 1 Introduction Most of the recent question answering benchmarks require systems to pinpoint the span of the answer to the question in the given text. In the well-known SQuAD 2.0 dataset (Rajpurkar et al., 2018), sys- tems are able to extract the correct answer span in the following paragraph as an answer to the ques- tion: “What tactic did researchers employ to offset the former deficit of work surrounding the complex- ity of algorithmic problems?”: “Before the actual research explicitly devoted to the complexity of algorithmic problems started off, numerous foundations were laid out by various researchers. Most influential among these was the definition of Turing machines by Alan Turing in 1936, which turned out to be a very robust and flexible simplification of a computer.” (sample from Rajpurkar et al. (2018)) Results of state-of-the-art (SOTA) systems on these datasets have reached Exact Match (EM) and F1 performances of 90.9 and 93.2, respectively 1. Some of these models even outperform the human baseline (which lies at F1 = 89.4 for SQuAD 2.0). It is unclear to which extent the existing bench- marks actually require systems to comprehend texts and to what extent these systems rely on surface cues signalling a match between question and an- swer span. Most state-of-the-art models rely on transformer models such as BERT and ALBERT (Lan et al., 2020) that are pre-trained on supple- mentary tasks using large amounts of textual data (Devlin et al., 2019) and employ extensive self- attention mechanisms that have been shown to learn many of the features of a classic natural language processing (NLP) pipeline (Tenney et al., 2019). It has been shown for a number of …

Content truncated to 500 words