Perhaps PTLMs Should Go to School – A Task to Assess Open Book and Closed Book QA

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6104–6111 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6104 Perhaps PTLMs Should Go to School – A Task to Assess Open Book and Closed Book QA Manuel R. Ciosici, Joe Cecil, Alex Hedges, Dong-Ho Lee, Marjorie Freedman, Ralph Weischedel manuelc@isi.edu, mrf@isi.edu, weisched@isi.edu Information Sciences Institute, University of Southern California Abstract Our goal is to deliver a new task and leader- board to stimulate research on question answer- ing and pre-trained language models (PTLMs) to understand a significant instructional docu- ment, e.g., an introductory college textbook or a manual. PTLMs have shown great success in many question-answering tasks, given signif- icant supervised training, but much less so in zero-shot settings. We propose a new task that includes two college-level introductory texts in the social sciences (American Government 2e) and humanities (U.S. History), hundreds of true/false statements based on review ques- tions written by the textbook authors, valida- tion/development tests based on the first eight chapters of the textbooks, blind tests based on the remaining textbook chapters, and baseline results given state-of-the-art PTLMs. Since the questions are balanced, random per- formance should be ~50%. T5, fine-tuned with BoolQ achieves the same performance, sug- gesting that the textbook’s content is not pre- represented in the PTLM. Taking the exam closed book, but having read the textbook (i.e., adding the textbook to T5’s pre-training), yields at best minor improvement (56%), suggesting that the PTLM may not have “understood” the textbook (or perhaps misunderstood the ques- tions). Performance is better (~60%) when the exam is taken open-book (i.e., allowing the ma- chine to automatically retrieve a paragraph and use it to answer the question). 1 Introduction Question answering (QA) is a yardstick for measur- ing machine understanding performance (Hermann et al., 2015). QA’s popularity as an evaluation tech- nique has led to several sub-categories: tasks can require a model to answer questions from either its background knowledge or from a short passage (e.g., SQuAD, Rajpurkar et al., 2016) or with infor- mation retrieval to allow the model to search for the answer in a large corpus (e.g., ARC, Clark et al., 2018). Answering can take the form of true/false classification (BoolQ, Clark et al., 2019), multiple- choice, span selection (SQuAD, Rajpurkar et al., 2016), or text generation (TriviaQA, Joshi et al., 2017). Transformer architectures optimized for specific QA formulations have driven recent progress in question answering. For example, some models tar- get IR-oriented QA (Guu et al., 2020) while others optimize their learning strategy to specific question types (e.g., by optimizing for expected answers to factoid questions, Roberts et al., 2020). While spe- cialization improves performance, it limits general- ization. UnifiedQA (Khashabi et al., 2020) takes a step forward by generalizing the architecture and training over multiple data sets with different QA formulations. Most research assumes that the information nec- essary to answer questions is either included with the query (e.g., BoolQ, SQuAD 1.1) or that the information was already stored …

Content truncated to 500 words