MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6521–6532, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6521 MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics Anthony Chen UC Irvine anthony.chen@uci.edu Gabriel Stanovsky∗ The Hebrew University gabis@cse.huji.ac.il Sameer Singh UC Irvine sameer@uci.edu Matt Gardner AI2 Irvine mattg@allenai.org Abstract Posing reading comprehension as a genera- tion problem provides a great deal of flexibil- ity, allowing for open-ended questions with few restrictions on possible answers. How- ever, progress is impeded by existing gen- eration metrics, which rely on token over- lap and are agnostic to the nuances of read- ing comprehension. To address this, we in- troduce a benchmark for training and evalu- ating generative reading comprehension met- rics: MOdeling Correctness with Human Annotations. MOCHA contains 40K human judgement scores on model outputs from 6 di- verse question answering datasets and an addi- tional set of minimal pairs for evaluation. Us- ing MOCHA, we train a Learned Evaluation metric for Reading Comprehension, LERC, to mimic human judgement scores. LERC outperforms baseline metrics by 10 to 36 absolute Pearson points on held-out annota- tions. When we evaluate robustness on mini- mal pairs, LERC achieves 80% accuracy, out- performing baselines by 14 to 26 absolute per- centage points while leaving significant room for improvement. MOCHA presents a chal- lenging problem for developing accurate and robust generative reading comprehension met- rics.1 1 Introduction Reading comprehension (RC) has seen significant progress in the last few years, with a number of question answering (QA) datasets being created (Rajpurkar et al., 2016; Lai et al., 2017; Talmor et al., 2018). However, a majority of datasets are presented using a span-selection or multiple-choice (MC) format. Both formats are easy to evaluate, ∗Work done while at the Allen Institute for AI and the University of Washington. 1The dataset, code, a leaderboard, and a demo are available at https://allennlp.org/mocha. Passage: ...Behind one door is a lady whom the king has deemed an appropriate match for the accused; behind the other is a fierce, hungry tiger. Both doors are heavily soundproofed to prevent the accused from hearing what is behind each one. . . Question: What feature do the doors have? Reference: soundproofed Candidate: They are heavily soundproofed to prevent the accused from hearing what’s behind each one. Human Judgement: 5 out of 5 LERC: 4.98 out of 5 BLEU-1: 0.07 ROUGE-L: 0.15 METEOR: 0.17 Figure 1: Generative reading comprehension example. Properly scoring the candidate requires access to the passage. Current metrics, such as BLEU, ROUGE and METEOR, are agnostic to the end-task while LERC is trained with the passage and question as input. As a result, LERC assigns a score that better reflects human judgement. but in return, have restrictions placed on the ques- tions that can be asked or the answers that can be returned. Furthermore, both formats hinge on dis- tractor spans/choices for learning to be effective. Ensuring high quality distractors is a challenging task in and of itself, which can …

Content truncated to 500 words