Information Seeking in the Spirit of Learning: A Dataset for Conversational Curiosity

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8153–8172, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8153 Information Seeking in the Spirit of Learning: A Dataset for Conversational Curiosity Pedro Rodriguez ∗ Computer Science University of Maryland pedro@cs.umd.edu Paul Crook Facebook pacrook@fb.com Seungwhan Moon Facebook shanemoon@fb.com Zhiguang Wang Facebook zgwang@fb.com Abstract Open-ended human learning and information- seeking are increasingly mediated by digi- tal assistants. However, such systems of- ten ignore the user’s pre-existing knowledge. Assuming a correlation between engagement and user responses such as “liking” messages or asking followup questions, we design a Wizard-of-Oz dialog task that tests the hypoth- esis that engagement increases when users are presented with facts related to what they know. Through crowd-sourcing of this experiment, we collect and release 14K dialogs (181K ut- terances) where users and assistants converse about geographic topics like geopolitical enti- ties and locations. This dataset is annotated with pre-existing user knowledge, message- level dialog acts, grounding to Wikipedia, and user reactions to messages. Responses us- ing a user’s prior knowledge increase engage- ment. We incorporate this knowledge into a multi-task model that reproduces human assis- tant policies and improves over a BERT content model by 13 mean reciprocal rank points. 1 Introduction Conversational agents such as Alexa, Siri, and Google Assistant should help users discover, learn, and retain novel factual information. More gen- erally, systems for conversational information- seeking should help users develop their information need, be mixed-initiative, incorporate user memory, and reason about the utility of retrieved informa- tion as a combined set (Radlinski and Craswell, 2017). We focus on a curiosity-driven, information- seeking scenario where a user starts a conversation with an assistant by asking an open-ended question and then drills down into interest areas (Figure 1). In this setting, what policies should assistants pursue to maintain the user’s interest in the topic? ∗⋆Work done while interning at Facebook. U: <assistant wake-word>, tell me about Tahiti. A: It’s the largest island in French Polynesia, near the center of the Pacific U: What is its history with France? Figure 1: An example of information-seeking dialog that the Curiosity dataset aims to support. Assistants should answer user questions and convey information that inspires meaningful followup questions. Theories of human learning, such as Vygotsky’s zone of proximal development, propose that learn- ing novel information should be rooted in pre- existing knowledge and skills of the learner (Chaik- lin, 2003). Considering this, a good policy may give general information about Tahiti; a better pol- icy would select information related to the user’s knowledge (e.g., familiarity with France). We hypothesize that engagement is correlated with policies that integrate a user’s pre-existing knowl- edge, and test this through a large-scale, Wizard- of-Oz (WoZ) style collection (Kelley, 1984; Wen et al., 2017) that captures assistant policies, user re- actions, and topically relevant entities that the user knows about. The Curiosity dataset has 14,048 En- glish dialogs annotated with sentence-level knowl- edge grounding, the user’s prior knowledge, dialog acts per …

Content truncated to 500 words