Asking Clarification Questions in Knowledge-Based Question Answering

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1618–1629, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1618 Asking Clarification Questions in Knowledge-Based Question Answering Jingjing Xu1∗, Yuechen Wang2, Duyu Tang3, Nan Duan3, Pengcheng Yang1, Qi Zeng1, Ming Zhou3, Xu Sun1 1 MOE Key Lab of Computational Linguistics, School of EECS, Peking University 2 University of Science and Technology of China 3 Microsoft Research Asia, Beijing, China {jingjingxu,yang pc,pkuzengqi,xusun}@pku.edu.cn wyc9725@mail.ustc.edu.cn {dutang,nanduan,mingzhou}@microsoft.com Abstract The ability to ask clarification questions is essential for knowledge-based question an- swering (KBQA) systems, especially for han- dling ambiguous phenomena. Despite its importance, clarification has not been well explored in current KBQA systems. Fur- ther progress requires supervised resources for training and evaluation, and powerful models for clarification-related text understanding and generation. In this paper, we construct a new clarification dataset, CLAQUA, with nearly 40K open-domain examples. The dataset sup- ports three serial tasks: given a question, iden- tify whether clarification is needed; if yes, generate a clarification question; then predict answers base on external user feedback. We provide representative baselines for these tasks and further introduce a coarse-to-fine model for clarification question generation. Experi- ments show that the proposed model achieves better performance than strong baselines. The further analysis demonstrates that our dataset brings new challenges and there still remain several unsolved problems, like reasonable automatic evaluation metrics for clarification question generation and powerful models for handling entity sparsity.1 1 Introduction Clarification is an essential ability for knowledge- based question answering, especially when han- dling ambiguous questions (Demetras et al., 1986). In real-world scenarios, ambiguity is a common phenomenon as many questions are not clearly articulated, e.g. “What are the languages used to create the source code of Midori?” in Fig- ure 1. There are two “Midori” using different pro- gramming languages, which confuses the system. ∗ The work was done while Jingjing Xu and Yuechen Wang were interns in Microsoft Research, Asia. 1The dataset and code will be released at https:// github.com/msra-nlc/MSParS_V2.0 What are the languages used to create the source code of Midori? I mean the first one. When you say the source code language used in the program Midori, are you referring to web browser Midori or operating system Midori? C. Midori Type Description computer.web_browser computer.software… Midori is a free and open- source light-weight web browser.... Midori Type Description computer.operating_system computer.software… Midori was the code name for a managed code operating system being developed… Language C Language M# Figure 1: An example of a clarification question in KBQA. There are two entities named “Midori” using different programming languages, which confuses the system. For these ambiguous or confusing questions, it is hard to directly give satisfactory responses unless systems can ask clarification questions to confirm the participant’s intention. Therefore, this work explores how to use clarification to improve cur- rent KBQA systems. pus,We introduce an open-domain clarification cor- CLAQUA, for KBQA. Unlike previous clarification-related datasets with …

Content truncated to 500 words