R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6740–6750 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6740 R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason Naoya Inoue1,2 Pontus Stenetorp2,3 Kentaro Inui1,2 1Tohoku University 2RIKEN 3University College London {naoya-i, inui}@ecei.tohoku.ac.jp p.stenetorp@cs.ucl.ac.uk Abstract Recent studies have revealed that reading com- prehension (RC) systems learn to exploit an- notation artifacts and other biases in current datasets. This prevents the community from reliably measuring the progress of RC systems. To address this issue, we introduce R4C, a new task for evaluating RC systems’ internal reasoning. R4C requires giving not only an- swers but also derivations: explanations that justify predicted answers. We present a reli- able, crowdsourced framework for scalably an- notating RC datasets with derivations. We cre- ate and publicly release the R4C dataset, the first, quality-assured dataset consisting of 4.6k questions, each of which is annotated with 3 reference derivations (i.e. 13.8k derivations). Experiments show that our automatic evalua- tion metrics using multiple reference deriva- tions are reliable, and that R4C assesses dif- ferent skills from an existing benchmark. 1 Introduction Reading comprehension (RC) has become a key benchmark for natural language understanding (NLU) systems, and a large number of datasets are now available (Welbl et al., 2018; Koˇcisk`y et al., 2018; Yang et al., 2018, i.a.). However, it has been established that these datasets suffer from annota- tion artifacts and other biases, which may allow systems to “cheat”: Instead of learning to read and comprehend texts in their entirety, systems learn to exploit these biases and find answers via sim- ple heuristics, such as looking for an entity with a particular semantic type (Sugawara et al., 2018; Mudrakarta et al., 2018) (e.g. given a question start- ing with Who, a system finds a person entity found in a document). To address this issue, the community has intro- duced increasingly more difficult Question Answer- ing (QA) problems, for example, so that answer- Title: Return to Olympus [1] Return to Olympus is the only album by the alternative rock band Malfunkshun. [2] It was released after the band had broken up and after lead singer Andrew Wood (later of Mother Love Bone) had died... [3] Stone Gossard had compiled… Title: Mother Love Bone [4] Mother Love Bone was an American rock band that… [5] The band was active from… [6] Frontman Andrew Wood’s personality and compositions helped to catapult the group to... [7] Wood died only days before the scheduled release of the band’s debut album, “Apple”, thus ending the… [Malfunkshun] is [a rock band] [Andrew Wood] is lead singer of [Malfunkshun] [Andrew Wood] died just before the release of [Apple] [Andrew Wood] is a member of [Mother Love Bone] [Malfunkshun] is former of [Mother Love Bone] R4C: Derivation Supporting facts (SFs): [1], [2], [4], [6], [7] Malfunkshun Question What was the former band of the member of Mother Love Bone who died just before the release of …

Content truncated to 500 words