SCDE: Sentence Cloze Dataset with High Quality Distractors From Examinations

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5668–5683 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5668 SCDE: Sentence Cloze Dataset with High Quality Distractors From Examinations Xiang Kong∗, Varun Gangal∗, Eduard Hovy Language Technologies Institute Carnegie Mellon University {xiangk,vgangal,hovy}@cs.cmu.edu Abstract We introduce SCDE, a dataset to evaluate the performance of computational models through sentence prediction. SCDE is a human- created sentence cloze dataset, collected from public school English examinations. Our task requires a model to fill up multiple blanks in a passage from a shared candidate set with distractors designed by English teachers. Ex- perimental results demonstrate that this task requires the use of non-local, discourse-level context beyond the immediate sentence neigh- borhood. The blanks require joint solving and significantly impair each other’s context. Fur- thermore, through ablations, we show that the distractors are of high quality and make the task more challenging. Our experiments show that there is a significant performance gap be- tween advanced models (72%) and humans (87%), encouraging future models to bridge this gap.1 2 1 Introduction Cloze questions were first proposed by Taylor (1953) as a readability test, motivated by Gestalt psychology. They become an efficient way of testing reading for public exams, overtaking the dominant paradigm of subjective questions (Fo- tos, 1991; Jonz, 1991). Cloze datasets (Zweig and Burges, 2011; Hermann et al., 2015; Hill et al., 2015; Paperno et al., 2016; Onishi et al., 2016; Xie et al., 2018) became prevalent as question- answering (QA) benchmarks since they are con- venient either to be generated automatically or by annotators. These datasets could be split into two clear types: 1. Where the context is a complete text, and there is an explicit question posed which is a statement with a cloze gap. The an- swer is either generated freely or is a span ∗ Equal Contribution 1Data: vgtomahawk.github.io/sced.html 2Code: https://github.com/shawnkx/SCDE from the context, e.g. Children’s Books Test (CBT) (Hill et al., 2015). 2. Where the context itself comes with cloze gaps. There is no explicit question. The an- swer is generated freely or chosen from a set of candidates, e.g. CLOTH (Xie et al., 2018). Herein, we focus on the 2nd category. A com- mon property of these datasets is that they have gaps at the level of words, entities or short syntac- tic spans. The entity and span-based clozes may sometimes be multi-token, but they do not extend beyond a few tokens. Nevertheless, none of these datasets have cloze gaps at the level of full sen- tences. Since many syntactic and semantic cues are present in the same sentence, this makes the gap easier to fill compared to the sentence level cloze case where models would have to rely on “discourse” cues beyond the same sentence. Besides lack of intra-sentence cues, sentence- level cloze may require comparing candidates of very different lengths. For instance, the example in Table 1 has a standard deviation of 7.6 with candidate lengths between 3 to 25. A model …

Content truncated to 500 words