A Surprisingly Robust Trick for the Winograd Schema Challenge

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4837–4842 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4837 A Surprisingly Robust Trick for the Winograd Schema Challenge Vid Kocijan1, Ana-Maria Cret¸u2, Oana-Maria Camburu1,3, Yordan Yordanov1, Thomas Lukasiewicz1,3 1University of Oxford 2Imperial College London 3Alan Turing Institute, London firstname.lastname@cs.ox.ac.uk, a.cretu@imperial.ac.uk Abstract The Winograd Schema Challenge (WSC) da- taset WSC273 and its inference counterpart WNLI are popular benchmarks for natural lan- guage understanding and commonsense rea- soning. In this paper, we show that the perfor- mance of three language models on WSC273 consistently and robustly improves when fine- tuned on a similar pronoun disambiguation problem dataset (denoted WSCR). We addi- tionally generate a large unsupervised WSC- like dataset. By fine-tuning the BERT lan- guage model both on the introduced and on the WSCR dataset, we achieve overall accu- racies of 72.5% and 74.7% on WSC273 and WNLI, improving the previous state-of-the- art solutions by 8.8% and 9.6%, respectively. Furthermore, our fine-tuned models are also consistently more accurate on the “complex” subsets of WSC273, introduced by Trichelair et al. (2018). 1 Introduction The Winograd Schema Challenge (WSC) (Leves- que et al., 2012, 2011) was introduced for testing AI agents for commonsense knowledge. Here, we refer to the most popular collection of such sen- tences as WSC273, to avoid confusion with other slightly modified datasets, such as PDP60, (Davis et al., 2017) and the Definite Pronoun Resolution dataset (Rahman and Ng, 2012), denoted WSCR in the sequel. WSC273 consists of 273 instan- ces of the pronoun disambiguation problem (PDP) (Morgenstern et al., 2016). Each is a sentence (or two) with a pronoun referring to one of the two or more nouns; the goal is to predict the correct one. The task is challenging, since WSC examples are constructed to require human-like commonsense knowledge and reasoning. The best known solu- tions use deep learning with an accuracy of 63.7% (Opitz and Frank, 2018; Trinh and Le, 2018). The problem is difficult to solve not only because of the commonsense reasoning challenge, but also due to the small existing datasets making it difficult to train neural networks directly on the task. Neural networks have proven highly effective in natural language processing (NLP) tasks, out- performing other machine learning methods and even matching human performance (Hassan et al., 2018; Nangia and Bowman, 2018). However, su- pervised models require many per-task annotated training examples for a good performance. For tasks with scarce data, transfer learning is often applied (Howard and Ruder, 2018; Johnson and Zhang, 2017), i.e., a model that is already trained on one NLP task is used as a starting point for other NLP tasks. A common approach to transfer learning in NLP is to train a language model (LM) on large amounts of unsupervised text (Howard and Ruder, 2018) and use it, with or without further fine-tu- ning, to solve other downstream tasks. Build- ing on top of a LM has proven to be very …

Content truncated to 500 words