Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7329–7346 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7329 Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering Siddhant Garg and Alessandro Moschitti Amazon Alexa AI {sidgarg,amosch}@amazon.com Abstract In this paper we propose a novel approach towards improving the efficiency of Question Answering (QA) systems by filtering out ques- tions that will not be answered by them. This is based on an interesting new finding: the an- swer confidence scores of state-of-the-art QA systems can be approximated well by mod- els solely using the input question text. This enables preemptive filtering of questions that are not answered by the system due to their answer confidence scores being lower than the system threshold. Specifically, we learn Transformer-based question models by distill- ing Transformer-based answering models. Our experiments on three popular QA datasets and one industrial QA benchmark demonstrate the ability of our question models to approximate the Precision/Recall curves of the target QA system well. These question models, when used as filters, can effectively trade off lower computation cost of QA systems for lower Recall, e.g., reducing computation by ∼60%, while only losing ∼3−4% of Recall. 1 Introduction Question Answering (QA) technology is at the core of several commercial applications, e.g., virtual assistants such as Alexa, Google Home and Siri, serving millions of users. Optimizing the efficiency of such systems is vital to reduce their operational costs. Recently, there has been a large body of re- search devoted towards reducing the compute com- plexity of retrieval (Gallagher et al., 2019; Tan et al., 2019) and transformer-based QA models (Sanh et al., 2019; Soldaini and Moschitti, 2020). An alternate solution for improving QA system efficiency aims to discard questions that will most probably be incorrectly answered by the system, us- ing automatic classifiers. For example, Fader et al. (2013); Faruqui and Das (2018) aim to capture the grammaticality and well-formedness of questions. However, these methods do not take the specific answering ability of the target system into account. In practice, QA systems typically do not answer a significant portion of user questions since their answer scores could be lower than a confidence threshold (Kamath et al., 2020), tuned by the sys- tem to achieve the required Precision. For example, QA systems for medical domains exhibit a high Precision since providing incorrect/imprecise an- swers can have critical consequences for the end user. Based on the above rationale, discarding ques- tions that will not be answered by the QA sys- tem presents a remarkable cost-saving opportunity. However, applying this idea may appear unrealistic from the outset since the QA system must first be executed to generate the answer confidence score. In this paper, we take a new perspective on im- proving QA system efficiency by preemptively fil- tering out questions that will not be answered by the system, by means of a question filtering model. This is based on our interesting finding that the …

Content truncated to 500 words