Turning silver into gold: error-focused corpus reannotation with active learning

Proceedings of Recent Advances in Natural Language Processing, pages 758–767, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_088 758 Turning Silver into Gold: Error-Focused Corpus Reannotation with Active Learning Pierre Andr´e M´enard, Antoine Mougeot Centre de recherche informatique de Montr´eal pierre-andre.menard@crim.ca, antoine.mougeot@crim.ca Abstract While high quality gold standard anno- tated corpora are crucial for most tasks in natural language processing, many an- notated corpora published in recent years, created by annotators or tools, contains noisy annotations. These corpora can be viewed as more silver than gold stan- dards, even if they are used in evaluation campaigns or to compare systems’ perfor- mances. As upgrading a silver corpus to gold level is still a challenge, we explore the application of active learning tech- niques to detect errors using four datasets designed for document classification and part-of-speech tagging. Our results show that the proposed method for the seeding step improves the chance of finding incorrect annotations by a factor of 2.73 when compared to random selec- tion, a 14.71% increase from the baseline methods. Our query method provides an increase in the error detection precision on average by a factor of 1.78 against random selection, an increase of 61.82% compared to other query approaches. 1 Introduction As machine learning is increasingly predominant in natural language processing tasks, the need for annotated data, and more specifically linguistic annotations, intensify greatly. Tasks like bias de- tection, named entity detection (NER) and recog- nition, part-of-speech (POS) tagging, semantic role labeling, assessment evaluation in discourse, dependency parsing and sentiment analysis can all be modeled as machine learning problems. They require a large amount of human annotated infor- mation to enrich raw textual resources. Creating large annotated resources for these tasks requires a lot of time and effort to insure a carefully planned and well-defined annotation protocol, obtain and encode expert knowledge, do curation steps and so on. Even when using highly qualified human an- notators, errors can always make their way into the final annotated corpus. Another way to obtain an- notations is to apply existing state-of-the-art algo- rithms (also trained on previously annotated data) on a raw corpus. While some researchers publish new versions of their hard earned annotated resources based on the feedback obtained from users, others often have to set them aside to work on other projects and resources, leaving the corpora with their original errors. One way to improve these resources with- out the same level of effort would be to reannotate them using an active learning process to quickly find errors and discrepancies and resubmit them to expert annotators. This article relates an experiment which ex- plores the potential application of active learning for corpus reannotation, which context is detailed in Section 2 with related works listed in Section 3. Two new approaches and multiple baselines are then explained in Section 4, followed by the list of datasets used (Section 5) and the experiments combining all these elements (Section 6). We then comment on the experiment (Section 7) and close with a review …

Content truncated to 500 words