A Survey on Asking Clarification Questions Datasets in Conversational Systems

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2698–2716 July 9-14, 2023 ©2023 Association for Computational Linguistics A Survey on Asking Clarification Questions Datasets in Conversational Systems Hossein A. Rahmani†∗ Xi Wang†∗ Yue Feng† Qiang Zhang‡ Emine Yilmaz† Aldo Lipani† †University College London, London, UK ‡Zhejiang University, Hangzhou, China {hossein.rahmani.22,xi-wang,yue.feng.20,emine.yilmaz,aldo.lipani}@ucl.ac.uk qiang.zhang.cs@zju.edu.cn Abstract The ability to understand a user’s underlying needs is critical for conversational systems, es- pecially with limited input from users in a con- versation. Thus, in such a domain, Asking Clar- ification Questions (ACQs) to reveal users’ true intent from their queries or utterances arise as an essential task. However, it is noticeable that a key limitation of the existing ACQs studies is their incomparability, from inconsistent use of data, distinct experimental setups and eval- uation strategies. Therefore, in this paper, to assist the development of ACQs techniques, we comprehensively analyse the current ACQs research status, which offers a detailed com- parison of publicly available datasets, and dis- cusses the applied evaluation metrics, joined with benchmarks for multiple ACQs-related tasks. In particular, given a thorough analy- sis of the ACQs task, we discuss a number of corresponding research directions for the inves- tigation of ACQs as well as the development of conversational systems. 1 Introduction Humans often resort to conversations and asking clarification questions to avoid misunderstandings when collaborating with others. Asking Clarifica- tion Questions (ACQs) is, therefore, a commonly used mechanism to boost efficiency on human- human as well as human-machine collaborative tasks (Shi et al., 2022; Zou et al., 2023; Shi et al., 2023; Feng et al., 2023). As an example of human- machine collaboration, conversational systems are developed to not only have a natural conversation with people but also to answer various questions of topics ranging from different domains (e.g., news, movie, and music) in an accurate and efficient manner (Gao et al., 2018). To effectively and effi- ciently answer various questions, it is essential for many existing conversational systems to capture ∗Equal Contribution people’s intents. Only then can conversational sys- tems accurately reply to a series of questions from users (Anand et al., 2020; Zamani et al., 2022). Nevertheless, one essential issue is that limited research exists on ACQs and most systems were trained with inconsistent and limited input of data resources. Indeed, in the literature, many studies introduced ACQs to assist conversational systems when applying to different / a mixture of domains (e.g., movie (Li et al., 2017) or open domain (Alian- nejadi et al., 2019)). There is also a lack of com- monly agreed benchmark datasets for the devel- opment of ACQs systems with comparable result analysis. However, on the other hand, in the lit- erature (Aliannejadi et al., 2019; Zamani et al., 2020; Kumar and Black, 2020; Feng et al., 2023), a growing number of studies released publicly avail- able datasets while showing a common interest in the ACQ research direction. This observed contra- diction leads to a necessity for a comprehensive …

Content truncated to 500 words