Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3309–3319 August 1–6, 2021. ©2021 Association for Computational Linguistics 3309 DISFL-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering Aditya Gupta♠ Jiacheng Xu♢∗ Shyam Upadhyay♠ Diyi Yang♣ Manaal Faruqui♠ ♠Google Assistant ♢The University of Texas at Austin ♣Georgia Institute of Technology disfl-qa@google.com Abstract Disfluencies is an under-studied topic in NLP, even though it is ubiquitous in human con- versation. This is largely due to the lack of datasets containing disfluencies. In this paper, we present a new challenge question answering dataset, DISFL-QA, a derivative of SQUAD, where humans introduce con- textual disfluencies in previously fluent ques- tions. DISFL-QA contains a variety of chal- lenging disfluencies that require a more com- prehensive understanding of the text than what was necessary in prior datasets. Experi- ments show that the performance of existing state-of-the-art question answering models de- grades significantly when tested on DISFL- QA in a zero-shot setting. We show data augmentation methods partially recover the loss in performance and also demonstrate the efficacy of using gold data for fine-tuning. We argue that we need large-scale disflu- ency datasets in order for NLP models to be robust to them. The dataset is pub- licly available at: https://github.com/ google-research-datasets/disfl-qa. 1 Introduction During conversations, humans do not always pre- meditate exactly what they are going to say; thus a natural conversation often includes interrup- tions like repetitions, restarts, or corrections. To- gether these phenomena are referred to as disflu- encies (Shriberg, 1994). Figure 1a shows different types of conventional disfluencies in an utterance, as described by Shriberg (1994). With the growing popularity of voice assistants, such disfluencies are of particular interest for goal- oriented or information seeking dialogue agents, because an NLU system, trained on fluent data, can easily get misled due to their presence. Fig- ure 1b shows how the presence of disfluencies in a ∗Work done during an internship at Google. Repetition When is Eas ugh Easter this year? Correction When is Lent I meant Easter this year? Restarts How much no wait when is Easter this year? (a) Conventional categories of Disfluencies. The reparan- dum (words intended to be corrected or ignored), inter- regnum (optional discourse cues) and repair are marked. Passage: The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (”Nor- man” comes from ”Norseman”) raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, . . . q1: In what country is Normandy located? dq1: In what country is Norse found no wait Normandy not Norse? T5(q1): France  T5(dq1): Denmark  q2: When were the Normans in Normandy? dq2: From which countries no tell me when were the Normans in Normandy? T5(q2): 10th and 11th centuries  T5(dq2): Denmark, Iceland and Norway  (b) Contextualized Disfluencies in DISFL-QA (§2). Figure 1: (a) Categories of disfluencies (Shriberg, 1994) (b) A passage and …

Content truncated to 500 words