Self-Supervised Test-Time Learning for Reading Comprehension

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1200–1211 June 6–11, 2021. ©2021 Association for Computational Linguistics 1200 Self-Supervised Test-Time Learning for Reading Comprehension Pratyay Banerjee Tejas Gokhale Arizona State UniversityChitta Baral pbanerj6, tgokhale, chitta@asu.edu Abstract Recent work on unsupervised question answer- ing has shown that models can be trained with procedurally generated question-answer pairs and can achieve performance competitive with supervised methods. In this work, we con- sider the task of unsupervised reading compre- hension and present a method that performs “test-time learning” (TTL) on a given con- text (text passage), without requiring training on large-scale human-authored datasets con- taining context-question-answer triplets. This method operates directly on a single test con- text, uses self-supervision to train models on synthetically generated question-answer pairs, and then infers answers to unseen human- authored questions for this context. Our method achieves accuracies competitive with fully supervised methods and significantly out- performs current unsupervised methods. TTL methods with a smaller model are also com- petitive with the current state-of-the-art in un- supervised reading comprehension. 1 Introduction Reading comprehension is the task in which sys- tems attempt to answer questions about a passage of text. Answers are typically found in the passage as text-spans or can be inferred through various forms of reasoning (Rajpurkar et al., 2016). The answer to the following question: “Who is the President of the United States?” depends on the timeframe and context of the pas- sage provided, and will be different for news ar- ticles written in 2001 vs. 2021. If the context is the script of the TV series “The West Wing”, the answer is “Jed Bartlet”, and even in this fictional setting, it will later change to “Matt Santos”. Knowledge sources such as Wikipedia get up- dated when new events occur (such as the outcome of elections), or new facts about the world are re- vealed (such as scientific discoveries), with con- tributors adding new information and removing information that is no longer valid (Almeida et al., 2007). With such context-dependent answers and continual changes in knowledge, it is hard to justify training models over fixed corpora for tasks such as question answering (QA). We would like models to answer questions based on the given context and not to learn biases from datasets or historical news articles. Moreover, supervised learning has been shown to perform poorly in QA tasks with adversarial ex- amples (Jia and Liang, 2017), domain shift (Jia and Liang, 2017; Yogatama et al., 2019; Kamath et al., 2020), and biased or imbalanced data (Agrawal et al., 2018; McCoy et al., 2019). For example, QA systems trained on Wikipedia fail to generalize to newer domains such as Natural Questions (Ren- nie et al., 2020) or biomedical data (Wiese et al., 2017), and suffer a significant drop in accuracy. Even small semantics-preserving changes to input sentences, such as the substitution of words by syn- onyms, have been shown to degrade performance in NLP tasks (Alzantot …

Content truncated to 500 words