Simulating Bandit Learning from User Feedback for Extractive Question Answering

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5167 - 5179 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Simulating Bandit Learning from User Feedback for Extractive Question Answering Ge Gao♦, Eunsol Choi♣ and Yoav Artzi♦ ♦Department of Computer Science and Cornell Tech, Cornell University ♣Department of Computer Science, The University of Texas at Austin ggao@cs.cornell.edu eunsol@utexas.edu yoav@cs.cornell.edu Abstract We study learning from user feedback for extractive question answering by simulating feedback using supervised data. We cast the problem as contextual bandit learning, and an- alyze the characteristics of several learning scenarios with focus on reducing data annota- tion. We show that systems initially trained on a small number of examples can dramatically improve given feedback from users on model- predicted answers, and that one can use exist- ing datasets to deploy systems in new domains without any annotation, but instead improving the system on-the-fly via user feedback. 1 Introduction Explicit feedback from users of NLP systems can be used to continually improve system perfor- mance. For example, a user posing a question to a question-answering (QA) system can mark if a predicted phrase is a valid answer given the con- text from which it was extracted. However, the dominant paradigm in NLP separates model train- ing from deployment, leaving models static follow- ing learning and throughout interaction with users. This approach misses opportunities for learning during system usage, which beside several excep- tions we discuss in Section 8 is understudied in NLP. In this paper, we study the potential of learn- ing from explicit user feedback for extractive QA through simulation studies. Extractive QA is a popular testbed for language reasoning, with rich prior work on datasets (e.g., Rajpurkar et al., 2016), task design (Yang et al., 2018; Choi et al., 2018), and model architecture de- velopment (Seo et al., 2017; Yu et al., 2018). Learn- ing from interaction with users remains relatively understudied, even though QA is well positioned to elicit user feedback. An extracted answer can be clearly visualized within its supporting context, and a language-proficient user can then easily validate Figure 1: Illustration of an interaction setup for learn- ing from user feedback for QA, and its potential. Given a user question, the system outputs an answer and high- lights it in its context. The user validates the answer given the context with binary feedback. We show per- formance progression from one of our online learning experiments on SQUAD with hand-crafted illustrative examples at two time steps. if the answer is supported or not.1 This allows for simple binary feedback, and creates a contextual bandit learning scenario (Auer et al., 2002; Lang- ford and Zhang, 2007). Figure 1 illustrates this learning signal and its potential. We simulate user feedback using several widely used QA datasets, and use it as a bandit signal for learning. We study the empirical characteristics of the learning process, including its performance, sensitivity to initial system performance, and trade- offs between online and …

Content truncated to 500 words