Active Learning for Assisted Corpus Construction: A Case Study in Knowledge Discovery from Biomedical Text

Proceedings of Recent Advances in Natural Language Processing, pages 216–225 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_026 216 Active Learning for Assisted Corpus Construction: A Case Study in Knowledge Discovery from Biomedical Text Hian Ca˜nizares-D´ıaz1, Alejandro Piad-Morffis1, Suilan Estevez-Velarde1, Yoan Guti´errez2, Yudivi´an Almeida Cruz1, Andres Montoyo2, and Rafael Mu˜noz-Guillena2 1School of Math and Computer Science, University of Havana, Cuba {hian.canizares,apiad,sestevez,estevanell,yudivian}@matcom.uh.cu 2University of Alicante, Spain {ygutierrez,montoyo,rafael}@dlsi.ua.es Abstract This paper presents an active learning ap- proach that aims to reduce the human effort required during the annotation of natural lan- guage corpora composed of entities and se- mantic relations. Our approach assists hu- man annotators by intelligently selecting the most informative sentences to annotate and then pre-annotating them with a few highly ac- curate entities and semantic relations. We de- fine an uncertainty-based query strategy with a weighted density factor, using similarity met- rics based on sentence embeddings. As a case study, we evaluate our approach via simulation in a biomedical corpus and estimate the poten- tial reduction in total annotation time. Experi- mental results suggest that the query strategy reduces by between 35% and 40% the num- ber of sentences that must be manually anno- tated to develop systems able to reach a tar- get F1 score, while the pre-annotation strat- egy produces an additional 24% reduction in the total annotation time. Overall, our prelimi- nary experiments suggest that as much as 60% of the annotation time could be saved while producing corpora that have the same useful- ness for training machine learning algorithms. An open-source computational tool that im- plements the aforementioned strategies is pre- sented and published online for the research community. 1 Introduction Machine learning, and specifically supervised learning, is one of the most effective tools for au- tomating complex cognitive tasks, such as recognis- ing objects in images or understanding natural lan- guage text. One of the main bottlenecks of super- vised learning is the need for high-quality datasets of labelled samples on which statistical models can be trained. These datasets are usually built by human experts in a lengthy and costly manual pro- cess. Active learning (Cohn, 2010) is an alternative paradigm to conventional supervised learning that has been proposed to reduce the costs involved in manual annotation . The key idea underlying active learning is that a learning algorithm can perform better with less training examples if it is allowed to actively se- lect which examples to learn from (Settles, 2009). In the supervised learning context, this paradigm changes the role of the human expert. In conven- tional supervised learning contexts, the human ex- pert guides the learning process by providing a large dataset of labelled examples. However, in active learning the active role is shifted to the al- gorithm and the human expert becomes an oracle, participating in a labelling-training-query loop. In the active paradigm, a model is incrementally built by training on a partial collection of samples and then selecting one or more unlabelled samples to query the human oracle for labels and increase the training …

Content truncated to 500 words