Building a Dataset for Automatically Learning to Detect Questions Requiring Clarification

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4701–4707 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4701 Building a Dataset for Automatically Learning to Detect Questions Requiring Clarification Ivano Lauriola, Kevin Small, Alessandro Moschitti Amazon Alexa Web Information Manhattan beach, CA, United States {lauivano, smakevin, amosch}@amazon.com Abstract Question Answering (QA) systems aim to return correct and concise answers in response to user questions. QA research generally assumes all questions are intelligible and unambiguous, which is unrealistic in practice as questions frequently encountered by virtual assistants are ambiguous or noisy. In this work, we propose to make QA systems more robust via the following two-step process: (1) classify if the input question is intelligible and (2) for such questions with contextual ambiguity, return a clarification question. We describe a new open-domain clarification corpus containing user questions sampled from Quora, which is useful for building machine learning approaches to solving these tasks. Keywords: Question Answering, Conversational, Clarification 1. Introduction QA is a widely-studied research area aimed at automat- ically generating a correct and concise answer to an in- put question. Over the past three decades, several QA corpora and corresponding machine learning methods have been developed. However, available datasets gen- erally focus on well-formed and understandable ques- tions such that an answer can be manually produced without additional context (Voorhees and Tice, 2000; Yang et al., 2015; Yao et al., 2013; Rajpurkar et al., 2016; Kwiatkowski et al., 2019). However, these datasets do not meet the requirements for building com- mercial virtual assistants, where the input traffic often includes unintelligible or ambiguous questions due to reasons including automatic speech recognition errors, questions requiring commonsense reasoning, and pe- culiarities of speech in a live conversation. After preliminary analyses of real user-generated traf- fic from a popular virtual assistant, we estimate that about 8.2% of the asked questions are unintelligible and more than 5% require clarification. Also, In our manual analysis we found that only a small percent- age of unintelligible or ambigiuous questions received an acceptable answer, and mainly this was obtained by chance. Given the importance of these questions in practice, existing works have proposed corpora to train models that can spot questions that are unintelligible or questions requiring clarification (QRC) and produce a crowd-sourced follow-up question for further informa- tion when appropriate. However, these works have lim- itations for commercial requirements including a single domain focus (Li et al., 2016; Rao and Daum´e, 2018), a generation process that does not reflect realistic use cases (Guo et al., 2017; Xu et al., 2019), or they are of limited size (De Boni and Manandhar, 2003; Stoy- anchev et al., 2014; Aliannejadi et al., 2019). A rel- evant resource is MIMICS (Zamani et al., 2020): a collection of search clarification datasets sampled from the Bing logs. However, our task significantly differs from those related MIMICS as (i) we consider ques- tions reflecting the interactions with a virtual assistant, which differ from queries used …

Content truncated to 500 words