Answering questions by learning to rank - Learning to rank by answering questions

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2531–2540, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2531 Abstract Answering multiple-choice questions in a setting in which no supporting documents are explicitly provided continues to stand as a core problem in natural language processing. The contribution of this article is two-fold. First, it describes a method which can be used to semantically rank documents extracted from Wikipedia or similar natural language corpora. Second, we propose a model employing the semantic ranking that holds the first place in two of the most popular leaderboards for answering multiple-choice questions: ARC Easy and Challenge. To achieve this, we introduce a self-attention based neural network that latently learns to rank documents by their importance related to a given question, whilst optimizing the objective of predicting the correct answer. These documents are considered relevant contexts for the underlying question. We have published the ranked documents so that they can be used off-the-shelf to improve downstream decision models. 1 Introduction The article at hand devotes to the problem of answering multiple-choice questions where the input consists of an inquiry expressed solely in raw natural language along with a small set of candidate answers (usually 4) from which only one is correct. Moreover, we are targeting questions from a field of science (e.g. chemistry, biology) as they are distinctly more challenging to answer than regular questions (Clark et al., 2018). A relevant example of such a question is the following: Which of the following is an example of a physical change? (A) Lighting a match (B) Breaking a glass (C) Burning of gasoline (D) Rusting of iron An important characteristic of all solutions developed for this task is that they are not given explicitly any external information in the form of documents supporting the correct answer or semi- structured information. However, external information is highly desirable, especially domain and common-sense knowledge. Thus, most of the state of the art solutions (Pîrtoacă et al., 2018; Nicula et al., 2018; Ni et al., 2018; Zhang et al., 2018) are using a two-step architecture, as shown in Figure 1. In the first phase, an information retrieval (IR) engine lexically searches for relevant supporting paragraphs in Wikipedia and other corpora considered relevant for the task. Having extracted some pertinent paragraphs (usually only one per candidate answer), potentially containing relevant information, a machine learning model is employed in the second step to reason about tuples (question, candidate answer, external information). Various downstream decision models are trained to infer whether the candidate answer is correct given the external information: transformers (Pan et al., 2019), attention models (Clark et al., 2018; Ni et al., 2018), or support vector machines (Clark et al., 2016). However, it has been previously reported that a retrieval-based engine alone is not able to return relevant documents from the reference corpora (Pîrtoacă et al., 2018; Zhang et al., 2018). Depending on the …

Content truncated to 500 words