Combining Cognitive Modeling and Reinforcement Learning for Clarification in Dialogue

Proceedings of the 28th International Conference on Computational Linguistics, pages 4417–4428 Barcelona, Spain (Online), December 8-13, 2020 4417 Combining Cognitive Modeling and Reinforcement Learning for Clarification in Dialogue Baber Khalid* Malihe Alikhani** Matthew Stone* Rutgers University* and University of Pittsburgh** firstname.lastname@rutgers.edu* malihe@pitt.edu** Abstract In many domains, dialogue systems need to work collaboratively with users to successfully re- construct the meaning the user had in mind. In this paper, we show how cognitive models of users’ communicative strategies can be leveraged in a reinforcement learning approach to dialogue planning to enable interactive systems to give targeted, effective feedback about the system’s understanding. We describe a prototype system that collaborates on reference tasks that distinguish arbitrarily varying color patches from similar distractors, and use experiments with crowd workers and analyses of our learned policies to document that our approach leads to context-sensitive clarification strategies that focus on key missing information, elicit correct answers that the system understands, and contribute to increasing dialogue success. 1 Introduction As dialogue systems move into richer domains, they must increasingly cope with situations where ob- jects don’t have generally-known names, where people describe concepts in incompatible ways, and where language use is fundamentally creative and uncertain (Furnas et al., 1987). People succeed in such situations thanks to their collaborative interactions, which let the speaker and the audience share responsibility for reaching a satisfactory mutual understanding (Clark, 1996). This paper contributes to a broader project of building dialogue systems that can do the same. We focus specifically on the problem of asking targeted, effective clarification questions with creative language. The difficulty of this problem is highlighted by the interactions shown in Figure 1, taken from the human-subjects evaluations reported in Section 6. In these dialogues, our system (playing the role of matcher in the referential communication task of Monroe et al. (2017)) uses varying clarification questions to follow up initial color descriptions provided by crowd workers (playing the role of director), and is thereby able to successfully distinguish the target color patches from their contextual distractors. x0 x1 x2 x0 x1 x2 director: lusher green matcher: sea green or bright green? director: sea green matcher: [Selects] director: light green matcher: Do you mean lime green? director: yes matcher: [Selects] Figure 1: Our system (matcher) interacting with crowd workers (director). System and users are pre- sented with the same color patches in random order; the director must identify x0 in a text chat conversa- tion. At left, the system’s clarification contrasts two alternative referents (lush, as it happens, is outside its vocabulary). At right, the system’s clarification redescribes the most likely alternative. The choice is based on a learned strategy, described in Section 5, that takes into account both the system’s uncertainty in context and likely patterns of user response. Both interactions here are successful. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 4418 In formulating such questions, the system must resolve the specific uncertainty it faces in context; there are …

Content truncated to 500 words