QA4IE: A Quality Assurance Tool for Information Extraction

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4497–4503 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4497 QA4IE: A Quality Assurance Tool for Information Extraction Rafael Jimenez Silva1, Kaushik Gedela1, Alex Marr1, Bart Desmet1, Carolyn Rosé1,2, Chunxiao Zhou1 1National Institutes of Health Clinical Center, Bethesda, Maryland, USA 2Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA {rafael.jimenezsilva, kaushik.gedela, alex.marr, bart.desmet, carolyn.rose, chunxiao.zhou}@nih.gov Abstract Quality assurance (QA) is an essential though underdeveloped part of the data annotation process. Although QA is supported to some extent in existing annotation tools, comprehensive support for QA is not standardly provided. In this paper we contribute QA4IE, a comprehensive QA tool for information extraction, which can (1) detect potential problems in text annotations in a timely manner, (2) accurately assess the quality of annotations, (3) visually display and summarize annotation discrepancies among annotation team members, (4) provide a comprehensive statistics report, and (5) support viewing of annotated documents interactively. This paper offers a competitive analysis comparing QA4IE and other popular annotation tools and demonstrates its features, usage, and effectiveness through a case study. The Python code, documentation, and demonstration video are available publicly at https://github.com/CC-RMD- EpiBio/QA4IE. Keywords: quality assurance, evaluation, discrepancy analysis 1. Introduction For the past 30 years, benchmark tasks have driven forward progress in the field of Natural Language Processing (NLP). As larger datasets have become the norm in the field, the number of available annotation tools has increased accordingly (Atdağ and Labatut, 2013; Beasley and Manda, 2018; Neves and Ševa, 2021). Though some benchmark tasks leverage training data with naturally occurring labels, common gold standard corpus construction requires substantial annotation effort. Despite advances in pipeline and model development, errors remain likely to occur. Therefore, there is a great need to develop tools that can help correct and improve annotated language resources. A comprehensive quality assurance tool helps guarantee the high quality of language resources and their downstream reliability for evaluation. In this paper we present QA4IE, a novel tool that addresses an important gap existing in other current annotation tools, namely comprehensive support for quality assurance. Due to the importance of QA in data annotation, most of the existing annotation tools include facilities for error checking (Cejuela et al., 2014; Cunningham et al., 2013; de Castilho et al., 2014; Grosman et al., 2020; Grundke et al., 2016; Klie et al., 2018; Kummerfeld, 2019; LightTag, 2021; Ogren, 2006; Stenetorp et al., 2012), though possibly perfunctory in nature. In contrast, QA4IE offers the following both for annotation and for information extraction: First, it offers automatic detection of potential problems in the annotation results. The user provides formatting standards that are used by QA4IE to validate the annotations and alert the user(s) to deviations from the standard. Second, QA4IE provides measures of agreement between annotators in terms of both token-level and entity- level IRR measurements. Third, based on its comparison between annotations across annotators, QA4IE provides a visual display of the discrepancies between annotators. Fourth, …

Content truncated to 500 words