Precise Task Formalization Matters in Winograd Schema Evaluations

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8275–8280, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8275 Precise Task Formalization Matters in Winograd Schema Evaluations Haokun Liu1∗ William Huang2∗ Dhara A. Mungra1 Samuel R. Bowman1,2,3 1Center for Data Science, New York University 2Courant Institute of Mathematical Sciences, New York University 3Department of Linguistics, New York University {haokunliu, wh629, dam797, bowman}@nyu.edu Abstract Performance on the Winograd Schema Chal- lenge (WSC), a respected English common- sense reasoning benchmark, recently rocketed from chance accuracy to 89% on the Super- GLUE leaderboard, with relatively little cor- roborating evidence of a correspondingly large improvement in reasoning ability. We hypoth- esize that much of this improvement comes from recent changes in task formalization— the combination of input specification, loss function, and reuse of pretrained parameters— by users of the dataset, rather than improve- ments in the pretrained model’s reasoning abil- ity. We perform an ablation on two Winograd Schema datasets that interpolates between the formalizations used before and after this surge, and find (i) framing the task as multiple choice improves performance by 2-6 points and (ii) several additional techniques, including the reuse of a pretrained language modeling head, can mitigate the model’s extreme sensitivity to hyperparameters. We urge future benchmark creators to impose additional structure to mini- mize the impact of formalization decisions on reported results. 1 Introduction Over the last couple of years, large pretrained mod- els have achieved human performance on a large share of established natural language understand- ing benchmark datasets (Devlin et al., 2019). Re- cent results report a surge in performance to near- human levels on the Winograd Schema Challenge (WSC; Levesque, 2011) in particular(Liu et al., 2019). However, variations in task formulation across papers and evaluations makes it hard to un- derstand the true degree of recent progress. The WSC is an English commonsense reason- ing evaluation that requires a model to resolve ∗Equal contribution. carefully-constructed ambiguous pronouns. For example, in the sentence “Jim yelled at Kevin be- cause he was so upset.” the reader will likely have to consider the motivation of the query noun phrase (NP) to recognize whether the pronoun he refers to Jim. The accuracy of WSC has seen an abrupt in- crease from 64% to 89% on the SuperGLUE (Wang et al., 2019) leaderboard upon the release of RoBERTa (Liu et al., 2019). While many works (Kocijan et al., 2019; Liu et al., 2019; Raffel et al., 2020) attribute such improvements to improved pre- training and the use of auxiliary training datasets, the impact of the task formalization—the combi- nation of input specification, task specific layer de- sign, and loss function—has not yet been seriously studied. The SuperGLUE WSC baseline with BERT (64%) resolves pronoun references for individual examples by concatenating the pronoun and query NP embeddings and making a binary prediction for the NP span. Meanwhile, RoBERTa (89%) uses a pretrained masked language modeling (MLM) head as part of the output layer and treats the task as …

Content truncated to 500 words