Taking the best from the Crowd:Learning Question Passage Classification from Noisy Data

Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics (*SEM 2016), pages 136–141, Berlin, Germany, August 11-12, 2016. Taking the best from the Crowd: Learning Question Passage Classification from Noisy Data Azad Abad University of Trento azad.abad@unitn.it Alessandro Moschitti Qatar Computing Research Institute amoschitti@qf.org.qa Abstract In this paper, we propose methods to take into account the disagreement between crowd annotators as well as their skills for weighting instances in learning algo- rithms. The latter can thus better deal with noise in the annotation and produce higher accuracy. We created two passage rerank- ing datasets: one with crowdsource plat- form, and the second with an expert who completely revised the crowd annotation. Our experiments show that our weighting approach reduces noise improving passage reranking up to 1.47% and 1.85% on MRR and P@1, respectively. 1 Introduction One of the most important steps for building accu- rate QA systems is the selection/reranking of an- swer passage (AP) candidates typically provided by a search engine. This task requires the auto- matic learning of a ranking function, which pushes the correct answer passages (i.e., containing the answer to the question) higher in the list. The accuracy of such function, among other, also depends on the quality of the supervision pro- vided in the training data. Traditionally, the lat- ter is annotated by experts through a rather costly procedure. Thus, sometimes, only noisy annota- tions obtained via automatic labeling mechanisms are available. For example, the Text REtrieval Conference (TREC1) provides open-domain QA datasets, e.g., for factoid QA. This data contains a set of questions, the answer keywords and a set of unannotated candidate APs. The labeling of the latter can be automatically carried out by check- ing if a given passage contains the correct answer keyword or not. However, this method is prone to 1http://trec.nist.gov generate passage labels, i.e., containing the answer keyword but not supporting it. For instance, given the following question, Q, from TREC 2002-03 QA, associated with the answer key Denmark: Q: Where was Hans Christian Anderson born? the candidate passage: AP: Fairy Tales written by Hans Christian Ander- sen was published in 1835-1873 in Denmark. would be wrongly labeled as a correct passage since it contains Denmark. Such passages can be both misleading for training and unreliable for evaluating the reranking model, thus requiring manual annotation. Since the expert work is costly, we can rely on crowdsourcing platforms such as CrowdFlower2 for labeling data, faster and at lower cost (Snow et al., 2008). This method has shown promising results but it still produces noisy labels. Thus, a solution consists in (i) using redundant anno- tations from multiple annotators and (ii) resolv- ing their disagreements with a majority voting ap- proach (Sheng et al., 2008; Zhang et al., 2015). However, the consensus mechanism can still pro- duce annotation noise, which (i) depends on crowd workers’ skill and the difficulty of the given task; and (ii) can degrade the classifier accuracy. In this paper, we study methods to take into ac- …

Content truncated to 500 words