A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 897–911 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 897 A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation Jan Deriu1, Katsiaryna Mlynchyk1, Philippe Schl¨apfer1, Alvaro Rodrigo2, Dirk von Gr¨unigen1, Nicolas Kaiser1, Kurt Stockinger1, Eneko Agirre3, and Mark Cieliebak1 1Zurich University of Applied Sciences (ZHAW), Winterthur, Switzerland {deri, mlyn, scrp, vogr, stog, ciel}@zhaw.ch 2National Distance Education University (UNED), Madrid, Spain alvarory@lsi.uned.es 3University of the Basque Country (UPV/EHU), Donostia, Spain e.agirre@ehu.eus Abstract In this paper, we introduce a novel methodol- ogy to efficiently construct a corpus for ques- tion answering over structured data. For this, we introduce an intermediate representation that is based on the logical query plan in a database called Operation Trees (OT). This representation allows us to invert the annota- tion process without losing flexibility in the types of queries that we generate. Further- more, it allows for fine-grained alignment of query tokens to OT operations. In our method, we randomly generate OTs from a context-free grammar. Afterwards, an- notators have to write the appropriate natural language question that is represented by the OT. Finally, the annotators assign the tokens to the OT operations. We apply the method to create a new corpus OTTA (Operation Trees and Token Assignment), a large semantic pars- ing corpus for evaluating natural language in- terfaces to databases. We compare OTTA to Spider and LC-QuaD 2.0 and show that our methodology more than triples the annotation speed while maintaining the complexity of the queries. Finally, we train a state-of-the-art se- mantic parsing model on our data and show that our corpus is a challenging dataset and that the token alignment can be leveraged to increase the performance significantly. 1 Introduction Question Answering (QA) over structured data, also called Natural Language Interfaces to Databases (NLI2DB) or Text-to-SQL, is a key task in natural language processing and the semantic web. It is usually approached by mapping a natural language question (NL question) into executable queries in formal representations such as logical forms, SPARQL or SQL. The state-of-the-art in this problem uses machine learning techniques to learn the mapping. Unfortu- nately, the construction of labeled corpora to train and evaluate NLI2DB systems is time- and cost- intensive, which is slowing down progress in this area. In particular, it usually requires recruiting SQL or SPARQL experts to write queries for natu- ral language questions. For instance, in Spider (Yu et al., 2018), the authors recruited students to write SQL queries. They worked 500 person-hours to generate 5,600 queries, which corresponds to more than 5 minutes per question. As a more cost-effective alternative to writing formal queries manually, some authors propose to use templates to generate them automatically. For instance, LC-QUAD 2.0 (Dubey et al., 2019) used 22 templates based on the structure of the target knowledge graph. Constructing templates is also time-consuming, and the expressiveness of the automatically produced queries is limited. Apart from the high cost of generating queries, …

Content truncated to 500 words