Using Natural Language Relations between Answer Choices for Machine Comprehension

Proceedings of NAACL-HLT 2019, pages 4010–4015 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4010 Using Natural Language Relations between Answer Choices for Machine Comprehension Rajkumar Pujari Department of Computer Science Purdue University rpujari@purdue.edu Dan Goldwasser Department of Computer Science Purdue University dgoldwas@purdue.edu Abstract When evaluating an answer choice for Read- ing Comprehension task, other answer choices available for the question and the answers of related questions about the same paragraph of- ten provide valuable information. In this pa- per, we propose a method to leverage the nat- ural language relations between the answer choices, such as entailment and contradiction, to improve the performance of machine com- prehension. We use a stand-alone question an- swering (QA) system to perform QA task and a Natural Language Inference (NLI) system to identify the relations between the choice pairs. Then we perform inference using an Integer Linear Programming (ILP)-based rela- tional framework to re-evaluate the decisions made by the standalone QA system in light of the relations identified by the NLI system. We also propose a multitask learning model that learns both the tasks jointly. 1 Introduction Given an input text and a set of related questions with multiple answer choices, the reading com- prehension (RC) task evaluates the correctness of each answer choice. Current approaches to the RC task quantify the relationship between each ques- tion and answer choice independently and pick the highest scoring option. In this paper, we fol- low the observation that when humans approach such RC tasks, they tend to take a holistic view ensuring that their answers are consistent across the given questions and answer choices. In this work we attempt to model these pragmatic infer- ences, by leveraging the entailment and contradic- tion relations between the answer choices to im- prove machine comprehension. To help clarify these concepts, consider the following examples: How can the military benefit from the existence of the CIA? c1: They can use them c2: These agencies are keenly attentive to the military’s strategic and tactical requirements () c3: The CIA knows what intelligence the mili- tary requires and has the resources to obtain that intelligence () The above example contains multiple correct answer choices, some are easier to capture than others. For example, identifying that c3 is true might be easier than c2 based on its alignment with the input text. However, capturing that c3 entails c2 allows us to predict c2 correctly as well. Classification of the answer in red (marked ) could be corrected using the blue (marked ) answer choice. Q1: When were the eggs added to the pan to make the omelette? c1 c11: When they turned on the stove 2: When the pan was the right temperature () Q2: Why did they use stove to cook omelette? c2 c21: They didn’t use the stove but a microwave 2: Because they needed to heat up the pan () Similarly, answering Q1 correctly helps in an- swering Q2. Our goal is to leverage …

Content truncated to 500 words