The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2799–2829 May 2-6, 2023 ©2023 Association for Computational Linguistics The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents Xing Han Lu 1 2† Siva Reddy 1 2 3 Harm de Vries 2 1 Mila Quebec AI Institute, McGill University 2 ServiceNow Research 3 Facebook CIFAR AI Chair statcan.dialogue.dataset@mila.quebec Abstract We introduce the StatCan Dialogue Dataset1 consisting of 19,379 conversation turns be- tween agents working at Statistics Canada and online users looking for published data tables. The conversations stem from genuine intents, are held in English or French, and lead to agents retrieving one of over 5000 complex data tables. Based on this dataset, we propose two tasks: (1) automatic retrieval of relevant tables based on a on-going conversation, and (2) automatic generation of appropriate agent responses at each turn. We investigate the difficulty of each task by establishing strong baselines. Our ex- periments on a temporal data split reveal that all models struggle to generalize to future con- versations, as we observe a significant drop in performance across both tasks when we move from the validation to the test set. In addition, we find that response generation models strug- gle to decide when to return a table. Consider- ing that the tasks pose significant challenges to existing models, we encourage the community to develop models for our task, which can be directly used to help knowledge workers find relevant tables for live chat users. 1 Introduction One of the longstanding goals in Natural Lan- guage Processing (NLP) is to develop conversa- tional agents that assist people with concrete tasks, such as finding information in large collections of documents or booking restaurants and hotels. To aid the development of such virtual assistants, the research community is in need of benchmarks that reflect the intents and linguistic phenomena found in real-world applications. However, developing such real-world conversational datasets is challeng- ing in the current research landscape. On the one hand, academic labs often struggle to come up with †Work done as visiting researcher at ServiceNow Research 1Website: mcgill-nlp.github.io/statcan-dialogue-dataset U1: Hi, I’m looking to obtain quarterly data in regards to GDP grow (Canada), BC Housing STarts, Canada Oil Price/BBL A1: Hello, my name is Kelly C. Give me one moment as I search [...] A1: For GDP growth rates, please consult the following link: [...] A1: What do you mean by BC Housing Starts? U2: I’m required to research all of the housing starts for BC on a quarterly basis [...] U2: Housing starts are the number of new residential construction projects that have begun during any par- ticular month [...] A2: I would have monthly data regarding new building permits being issued [...] A2: Building permits, by type of structure and type of work: https[...] A2: I’ll have a look for oil prices. One moment. U3: Do you also have data to Canada’as oil Price/BBL ("WTI")? [...] A3: Are you looking for the …

Content truncated to 500 words