Reading Comprehension as Natural Language Inference:A Semantic Analysis

Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics (*SEM), pages 12–19 Barcelona, Spain (Online), December 12–13, 2020 12 Reading Comprehension as Natural Language Inference: A Semantic Analysis Anshuman Mishra ∗1, Dhruvesh Patel∗1, Aparna Vijayakumar∗1, Xiang Lorraine Li1, Pavan Kapanipathi2, and Kartik Talamadupula2 1 College of Information and Computer Sciences, University of Massachusetts Amherst 2IBM Research Abstract In the recent past, Natural language Inference (NLI) has gained significant attention, partic- ularly given its promise for downstream NLP tasks. However, its true impact is limited and has not been well studied. Therefore, in this paper, we explore the utility of NLI for one of the most prominent downstream tasks, viz. Question Answering (QA). We transform one of the largest available MRC dataset (RACE) to an NLI form, and compare the performances of a state-of-the-art model (RoBERTa) on both these forms. We propose new characteriza- tions of questions, and evaluate the perfor- mance of QA and NLI models on these cate- gories. We highlight clear categories for which the model is able to perform better when the data is presented in a coherent entailment form, and a structured question-answer concatena- tion form, respectively. 1 Introduction Given two sentences, a premise and a hypothesis, the task of Natural Language Inference (NLI) is to determine whether the premise entails the hypothe- sis or not. † The concept of semantic entailment is central to natural language understanding (Van Ben- them et al., 2008; MacCartney and Manning, 2009) and therefore, NLI models have been used to help with various downstream tasks like reading com- prehension (Trivedi et al., 2019), summarization (Falke et al., 2019; Kry´sci´nski et al., 2019), and dialog systems (Welleck et al., 2019). However, the performance of an NLI system on these down- ∗Equal contribution. This work is licensed under a Creative Commons Attri- bution 4.0 International License. License details: http: //creativecommons.org/licenses/by/4.0/. †The “not entailment” can further be subdivided into “neu- tral” and “contradiction”. However, we only use the two-class version of the problem in this work. stream tasks has not been studied with respect to semantic or reasoning categories. In this work, we use NLI to perform the task of multiple choice reading comprehension (MRC, or RC). We analyse the performance of an NLI model on this task through the lens of semantics by iden- tifying the reasoning categories (type of questions) where it is beneficial to use an NLI model. Drawing inspiration from the prior work in the area (Clark et al., 2018; Demszky et al., 2018; Trivedi et al., 2019), we use rule-based conversion to create an NLI version of the largest available RC dataset - RACE (Lai et al., 2017). We train a RoBERTa based RC model on the original dataset, and a similar RoBERTa based NLI model on the NLI version of the dataset. We evaluate and analyse the performance of both these models by character- izing the question types that are better suited for an NLI model and a QA model. 2 Related Work Reading comprehension …

Content truncated to 500 words