Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4473–4484 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4473 Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions Mohammad Aliannejadi1, Julia Kiseleva2, Aleksandr Chuklin3, Jeffrey Dalton4, Mikhail Burtsev5 1University of Amsterdam, 2Microsoft Research, 3Google Research 4University of Glasgow, 5MIPT & AIRI m.aliannejadi@uva.nl, julia.kiseleva@microsoft.com chuklin@google.com,jeff.dalton@glasgow.ac.uk,burtcev.ms@mipt.ru Abstract Enabling open-domain dialogue systems to ask clarifying questions when appropriate is an important direction for improving the qual- ity of the system response. Namely, for cases when a user request is not specific enough for a conversation system to provide an an- swer right away, it is desirable to ask a clarify- ing question to increase the chances of retriev- ing a satisfying answer. To address the prob- lem of ‘asking clarifying questions in open- domain dialogues’: (1) we collect and release a new dataset focused on open-domain single- and multi-turn conversations, (2) we bench- mark several state-of-the-art neural baselines, and (3) we propose a pipeline consisting of of- fline and online steps for evaluating the qual- ity of clarifying questions in various dialogues. These contributions are suitable as a founda- tion for further research. 1 Introduction The ultimate goal of a conversational system is to assist users by returning an appropriate answer in response to their requests (Kiseleva et al., 2016a; Li et al., 2021). Recent progress on neural ap- proaches to natural language processing (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020), and the availability of large amounts of conversational data have triggered a renaissance in end-to-end neu- ral open-domain chatbots (Adiwardana et al., 2020; Roller et al., 2021; Zhang et al., 2020; Burtsev et al., 2017; Dalton et al., 2020). There has been great progress on suggesting measures to evaluate what makes a conversation satisfying for users us- ing various human evaluation techniques (Li et al., 2019a; See et al., 2019b). Those efforts showed that suggested large pre-trained models do not always perform seamlessly (See et al., 2019a), and there are still several challenges needed to be solved for open-domain conversational systems (Huang et al., 2020). How to write a thank you letter after an interview? Are you interested in example of thank you letters for interview? Yes Here are a few examples: Tell me about source of the Nile Are you referring to the Nile river? No, the board game called the source of the Nile I'm looking for information on JAC chemical company [conversation continued] Are you interested in a specific type of product? Yes, surface treatment products that JAC chemical company manufactures Here is information about their products: (a) No need for clarification (b) Clarification question is needed to eliminate wrong answers (c) Clarification question is needed to return a right answer Figure 1: Examples of clarification questions embed- ded into the open domain conversations: (a) represents a clear user request, so clarification was unnecessary; (b) and (c) demonstrate a situation when the request is ambiguous and a system needs to …

Content truncated to 500 words