Learning a Cost-Effective Annotation Policy for Question Answering

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3051–3062, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3051 Learning a Cost-Effective Annotation Policy for Question Answering Bernhard Kratzwald♦ ♠ Stefan Feuerriegel♦ Huan Sun♠ ♦ Chair of Management Information Systems, ETH Zurich ♠ Department of Computer Science and Engineering, The Ohio State University {bkratzwald, sfeuerriegel}@ethz.ch sun.397@osu.edu Abstract State-of-the-art question answering (QA) re- lies upon large amounts of training data for which labeling is time consuming and thus ex- pensive. For this reason, customizing QA sys- tems is challenging. As a remedy, we propose a novel framework for annotating QA datasets that entails learning a cost-effective annota- tion policy and a semi-supervised annotation scheme. The latter reduces the human effort: it leverages the underlying QA system to suggest potential candidate annotations. Human anno- tators then simply provide binary feedback on these candidates. Our system is designed such that past annotations continuously improve the future performance and thus overall annotation cost. To the best of our knowledge, this is the first paper to address the problem of anno- tating questions with minimal annotation cost. We compare our framework against traditional manual annotations in an extensive set of ex- periments. We find that our approach can re- duce up to 21.1% of the annotation cost. 1 Introduction Question answering (QA) based on textual con- tent has attracted a great deal of attention in recent years (Chen et al., 2017; Lee et al., 2018, 2019; Xie et al., 2020). In order for state-of-the-art QA mod- els to succeed in real applications (e.g., customer service), there is often a need for large amounts of training data. However, manually annotating such data can be extremely costly. For example, in many realistic scenarios, there exists a list of questions from real users (e.g., search logs, FAQs, service-desk interactions). Yet, annotating such questions is highly expensive (Nguyen et al., 2016; He et al., 2018; Kwiatkowski et al., 2019): it re- quires the screening of a text corpus to find the rel- evant document(s) and subsequently screening the document(s) to identify the answering text span(s). Motivated by the above scenarios, we study cost-effective annotation for question answering, whereby we aim to accurately1 annotate a given set of user questions with as little cost as possible. Generally speaking, there has been extensive re- search on how to reduce effort in the process of data labeling (Haffari et al., 2009). For example, active learning for a variety of machine learning and NLP tasks (Siddhant and Lipton, 2018) aims to select a small, yet highly informative, subset of samples to be annotated. The selection of such samples is usually coupled with a particular model, and thus, the annotated samples may not necessarily help to improve a different model (Lowell et al., 2019). In contrast, we aim to annotate all given samples at low cost and in a manner that can subsequently be used to develop any advanced model. This is particularly relevant in the current era, where a …

Content truncated to 500 words