ClarQ: A large-scale and diverse dataset for Clarification Question Generation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7296–7301 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7296 ClarQ: A large-scale and diverse dataset for Clarification Question Generation Vaibhav Kumar Language Technologies Institute Carnegie Mellon University vaibhav2@cs.cmu.eu Alan W Black Language Technologies Institute Carnegie Mellon University awb@cs.cmu.edu Abstract Question answering and conversational sys- tems are often baffled and need help clari- fying certain ambiguities. However, limita- tions of existing datasets hinder the develop- ment of large-scale models capable of gener- ating and utilising clarification questions. In order to overcome these limitations, we de- vise a novel bootstrapping framework (based on self-supervision) that assists in the creation of a diverse, large-scale dataset of clarification questions based on post-comment tuples ex- tracted from stackexchange. The framework utilises a neural network based architecture for classifying clarification questions. It is a two-step method where the first aims to in- crease the precision of the classifier and sec- ond aims to increase its recall. We quantita- tively demonstrate the utility of the newly cre- ated dataset by applying it to the downstream task of question-answering. The final dataset, ClarQ, consists of ∼2M examples distributed across 173 domains of stackexchange. We re- lease this dataset1 in order to foster research into the field of clarification question genera- tion with the larger goal of enhancing dialog and question answering systems. 1 Introduction The ubiquitous nature of conversations has led to the identification of various interesting problems (Clark et al., 2019). One of these problems is the ability of a system to ask for clarifications (Rao and Daum´e III, 2018; Aliannejadi et al., 2019) to a natural language question. A user’s complex information need is often lost due to the brevity of the posed question. This leads to an under-specified question containing in- formation gaps which lowers the probability of providing the correct answer. Thus, it would be an improvement if a conversational or a question answering system had a mechanism for refining 1https://github.com/vaibhav4595/ClarQ user questions with follow-ups (De Boni and Man- andhar, 2003). In literature, such questions have been termed Clarification Questions (De Boni and Manandhar, 2003; Rao and Daum´e III, 2018, 2019). In the domain of question-answering, the major advantages of a clarification question are its ability to resolve ambiguities (Wang et al., 2018; Alian- nejadi et al., 2019) and to improve the probabil- ity of finding the most relevant answer. For con- versational systems, asking such questions help in driving the conversation deeper along with better engagement of the user (Li et al., 2016; Yu et al., 2016). Recently, Rao and Daum´e III (2018, 2019) have provided a dataset based on stackexchange and used it for clarification question retrieval as well as generation. They also modify a dataset based on Amazon Question-Answering and Product Re- views (McAuley et al., 2015; McAuley and Yang, 2016) to make it suitable for the same task. On the other hand, Aliannejadi et al. (2019) created a dataset (Qulac) built on top of …

Content truncated to 500 words