The TechQA Dataset

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1269–1278 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1269 The TechQA Dataset Vittorio Castelli1 Rishav Chakravarti2 Saswati Dana1 Anthony Ferritto2 Radu Florian1 Martin Franz 1 Dinesh Garg1 Dinesh Khandelwal1 Scott McCarley1 Michael McCawley3 Mohamed Nasr1 Lin Pan2 Cezar Pendus1 John Pitrelli1 Saurabh Pujar1 Salim Roukos1 Andrzej Sakrajda1 Avirup Sil1 Rosario Uceda-Sosa1 Todd Ward1 Rong Zhang1 1IBM Research AI, 2IBM Watson, 3 IBM Finance and Operations {vittorio, rchakravarti, raduf, franzm, jsmc, mmccawley, mnasr, panl, cpendus, pitrelli, roukos, ansa, avi, rosariou, toddward, zhangr}@us.ibm.com, {aferritto, saurabh.pujar}@ibm.com, {sadana04, garg.dinesh, dikhand1}@in.ibm.com Abstract We introduce TECHQA, a domain-adaptation question answering dataset for the technical support domain. The TECHQA corpus high- lights two real-world issues from the auto- mated customer support domain. First, it con- tains actual questions posed by users on a technical forum, rather than questions gen- erated specifically for a competition or a task. Second, it has a real-world size – 600 training, 310 dev, and 490 evaluation ques- tion/answer pairs – thus reflecting the cost of creating large labeled datasets with actual data. Hence, TECHQA is meant to stimulate re- search in domain adaptation rather than as a resource to build QA systems from scratch. TECHQA was obtained by crawling the IB- MDeveloper and DeveloperWorks forums for questions with accepted answers provided in an IBM Technote—a technical document that addresses a specific technical issue. We also release a collection of the 801,998 Technotes available on the web as of April 4, 2019 as a companion resource that can be used to learn representations of the IT domain language. 1 Introduction There is a tension between the development of novel capabilities in the early phases of the tech- nology lifecycle, using unlimited data and com- pute power, and the later development of practical solutions as that technology matures. The chal- lenges of creating practical solutions are twofold: developing robust, efficient algorithms and curat- ing appropriate training data. Here we describe the curation and public release of a dataset intended to further those algorithmic advances. The application domain is IT support, a no- table component of the trillion-dollar IT services industry1. We created a dataset using publicly available data: questions from technical forums and answers from technical documents, all in En- glish. We manually selected question-answer pairs that are appropriate for machine reading compre- hension techniques, and reserved questions where the answer is distributed across multiple separate spans or documents, and those that require rea- soning or substantial real world knowledge for future datasets. We release 600 questions for training purposes, of which 150 are not answer- able from the provided documents, as well as 160 answerable and 150 non-answerable ques- tions as development set. The blind test set con- tains 490 questions with similar answerable/non- answerable statistics to the development set. The purpose of the TECHQA dataset is to stimulate transfer learning research from popular question-answering scenarios—driven by large- scale open-domain datasets with short questions and answers—to a …

Content truncated to 500 words