DoQA - Accessing Domain-Specific FAQs via Conversational QA

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7302–7314 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7302 DoQA - Accessing Domain-Specific FAQs via Conversational QA Jon Ander Campos1, Arantxa Otegi1, Aitor Soroa1, Jan Deriu2, Mark Cieliebak2, Eneko Agirre1 1University of the Basque Country (UPV/EHU) 2Zurich University of Applied Sciences (ZHAW) 1 {jonander.campos, arantza.otegi, e.agirre, a.soroa}@ehu.eus 2{jan.deriu, mark.cieliebak}@zhaw.ch Abstract The goal of this work is to build conversa- tional Question Answering (QA) interfaces for the large body of domain-specific information available in FAQ sites. We present DoQA, a dataset with 2,437 dialogues and 10,917 QA pairs. The dialogues are collected from three Stack Exchange sites using the Wizard of Oz method with crowdsourcing. Compared to pre- vious work, DoQA comprises well-defined in- formation needs, leading to more coherent and natural conversations with less factoid ques- tions and is multi-domain. In addition, we in- troduce a more realistic information retrieval (IR) scenario where the system needs to find the answer in any of the FAQ documents. The results of an existing, strong, system show that, thanks to transfer learning from a Wikipedia QA dataset and fine tuning on a single FAQ domain, it is possible to build high quality con- versational QA systems for FAQs without in- domain training data. The good results carry over into the more challenging IR scenario. In both cases, there is still ample room for im- provement, as indicated by the higher human upperbound. 1 Introduction The overarching objective of our work is to access the large body of domain-specific information avail- able in Frequently Asked Question sites (FAQ for short) via conversational Question Answering (QA) systems. In particular, we want to know whether current techniques are able to work with limited training data, and without needing to gather data for each target FAQ domain. In this paper we present DoQA, a task and associated dataset for accessing domain-specific FAQs via conversational QA1. The dataset contains 2,437 information-seeking ques- tion/answer dialogues on three different domains 1The DoQA dataset is available here: http://ixa. eus/node/12931 Figure 1: A dialogue about cooking. On top, the orig- inal post, comprising a topic and an excerpt of the an- swer passage. In italics, dialogue acts (cf. Section 3). (10,917 questions in total). These dialogues are created using the Wizard of Oz technique by crowd- workers that play the following two roles: the user asks questions about a given topic posted in Stack Exchange2, and the domain expert replies to the questions by selecting a short span of text from the long textual reply in the original post. The first question is prompted by the real FAQ question, which sets the topic of interest driving the user questions. In addition to the extractive span, we also allow experts to rephrase it, in order to provide an abstractive, more natural, answer. The dataset covers unanswerable questions and some relevant dialogue acts. We focused on three different do- mains: Cooking, Travel and Movies. These forums 2https://stackexchange.com/ 7303 …

Content truncated to 500 words