Feedback Attribution for Counterfactual Bandit Learning in Multi-Domain Spoken Language Understanding

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1190–1198 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1190 Feedback Attribution for Counterfactual Bandit Learning in Multi-Domain Spoken Language Understanding Tobias Falke Amazon Alexa AI Berlin, Germany falket@amazon.com Patrick Lehnen Amazon Alexa AI Aachen, Germany plehnen@amazon.com Abstract With counterfactual bandit learning, models can be trained based on positive and negative feedback received for historical predictions, with no labeled data needed. Such feedback is often available in real-world dialog systems, however, the modularized architecture com- monly used in large-scale systems prevents the direct application of such algorithms. In this paper, we study the feedback attribution prob- lem that arises when using counterfactual ban- dit learning for multi-domain spoken language understanding. We introduce an experimental setup to simulate the problem on small-scale public datasets, propose attribution methods inspired by multi-agent reinforcement learning and evaluate them against multiple baselines. We find that while directly using overall feed- back leads to disastrous performance, our pro- posed attribution methods can allow training competitive models from user feedback. 1 Introduction Spoken language understanding (SLU) is a key component of task-oriented dialog systems (Tur and De Mori, 2011). It is commonly modeled as two tasks: Intent classification (IC), which assigns an intent to an utterance, and slot labeling (SL), which recognizes boundaries and types of slots in the utterance’s tokens. In recent years, neural models that jointly learn both tasks, in combination with pre-trained transformers (Chen et al., 2019; Zhang et al., 2019), have become the state-of-the- art approach to SLU (Louvan and Magnini, 2020; Weld et al., 2021). However, a sufficient amount of manually labeled data is needed for fine-tuning. If dialog systems are actively used, a cost- efficient alternative to manually labeled data is to leverage user feedback, which can appear explicitly (e.g. "no stop", "thank you") and implicitly (e.g. retries, interruptions). Such positive and negative feedback in response to a prediction is known as bandit feedback. Compared to ground truth labels it is less informative, as negative feedback does not reveal which prediction would have been better, but on the other hand, it is available in large quanti- ties without manual effort. Recent work proposed multiple possible ways in which such feedback can be used to improve dialog systems (Muralidharan et al., 2019; Ponnusamy et al., 2020; Kim and Kim, 2020; Falke et al., 2020). In this work, we focus on leveraging it via counterfactual bandit learning, a well-studied approach for training models with historical bandit feedback (Langford et al., 2008; Dudík et al., 2011; Joachims et al., 2018). The key challenge addressed in this paper is the feedback attribution problem arising in large- scale multi-domain SLU systems (see Figure 1). In such systems, a common architecture is to com- bine domain-specific IC and SL models with a do- main classifier (DC) (Jeong and Lee, 2009; Xu and Sarikaya, 2014; Hakkani-Tür et al., 2016), which allows to more easily update domain-specific be- havior regularly without being bottlenecked by a single …

Content truncated to 500 words