Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment

Findings of the Association for Computational Linguistics: ACL 2022, pages 926 - 937 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment Zichao Li1, Prakhar Sharma2, Xing Han Lu1, Jackie C.K. Cheung1, Siva Reddy1 1Mila, McGill University 2University of California, Los Angeles zichao.li@mila.quebec Abstract Most research on question answering focuses on the pre-deployment stage; i.e., building an accurate model for deployment. In this paper, we ask the question: Can we improve QA sys- tems further post-deployment based on user in- teractions? We focus on two kinds of improve- ments: 1) improving the QA system’s perfor- mance itself, and 2) providing the model with the ability to explain the correctness or incor- rectness of an answer. We collect a retrieval- based QA dataset, FEEDBACKQA, which con- tains interactive feedback from users. We col- lect this dataset by deploying a base QA sys- tem to crowdworkers who then engage with the system and provide feedback on the qual- ity of its answers. The feedback contains both structured ratings and unstructured natural lan- guage explanations. We train a neural model with this feedback data that can generate ex- planations and re-score answer candidates. We show that feedback data not only improves the accuracy of the deployed QA system but also other stronger non-deployed systems. The gen- erated explanations also help users make in- formed decisions about the correctness of an- swers.1 1 Introduction Much of the recent excitement in question answer- ing (QA) is in building high-performing models with carefully curated training datasets. Datasets like SQuAD (Rajpurkar et al., 2016), NaturalQues- tions (Kwiatkowski et al., 2019) and CoQA (Reddy et al., 2019) have enabled rapid progress in this area. Most existing work focuses on the pre-deployment stage; i.e., training the best QA model before it is released to users. However, this stage is only one stage in the potential lifecycle of a QA system. In particular, an untapped resource is the large amounts of user interaction data produced after the initial deployment of the system. Gathering this 1Project page: https://mcgill-nlp.github.io/feedbackqa/ data should in practice be relatively cheap, since users genuinely engage with QA systems (such as Google) for information needs and may provide feedback to improve their results.2 Exploiting this kind of user interaction data presents new research challenges, since they typ- ically consist of a variety of weak signals. For example, user clicks could indicate answer useful- ness (Joachims, 2002), users could give structured feedback in the form of ratings to indicate the use- fulness (Stiennon et al., 2020), or they could give unstructured feedback in natural language expla- nations on why an answer is correct or incorrect. User clicks have been widely studied in the field of information retrieval (Joachims, 2002). Here we study the usefulness of interactive feedback in the form of ratings and natural language explanations. Whilst there are different variants of QA tasks, this paper focuses primarily on retrieval-based QA (RQA; Chen et al. 2017; …

Content truncated to 500 words