Towards a Consensus Taxonomy for Annotating Errors in Automatically Generated Text

Proceedings of Recent Advances in Natural Language Processing, pages 527–540 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_058 527 Towards a Consensus Taxonomy for Annotating Errors in Automatically Generated Text Rudali Huidrom and Anya Belz ADAPT Research Centre Dublin City University Ireland {rudali.huidrom,anya.belz}@adaptcentre.ie Abstract Error analysis aims to provide insights into system errors at different levels of granular- ity. NLP as a field has a long-standing tradi- tion of analysing and reporting errors which is generally considered good practice. There are existing error taxonomies tailored for dif- ferent types of NLP task. In this paper, we report our work reviewing existing research on meaning/content error types in generated text, attempt to identify emerging consensus among existing meaning/content error taxonomies, and propose a standardised error taxonomy on this basis. We find that there is virtually complete agreement at the highest taxonomic level where errors of meaning/content divide into (1) Con- tent Omission, (2) Content Addition, and (3) Content Substitution. Consensus in the lower levels is less pronounced, but a compact stan- dardised consensus taxonomy can nevertheless be derived that works across generation tasks and application domains. 1 Introduction Error analysis and error type annotation are widely considered important for diverse natural language processing (NLP) tasks (Popovi´c and Burchardt, 2011; Costa et al., 2015). NLP has a long-standing track record in error analysis and error type an- notation (Macklovitch, 1991; Costa et al., 2015; Rivera-Trigueros, 2021), not only for directly im- proving system performance but also for providing guidance in improving evaluation methods. Errors of content (as opposed to errors of form such as grammatical or lexical-choice errors) are becoming more common in current language gener- ation outputs, given the growing dominance of neu- ral methods which are more prone to such errors than previous rule-based and statistical systems. Documenting and analysing what types of errors different systems make can help improve the se- mantic correctness (known as Adequacy in MT) of generated text. However, a large variety of different annotation schemes have been created (Huidrom and Belz, 2022), often task and/or domain-specific, which makes comparison between output annota- tions and thus incremental progress difficult. A standardised, task-agnostic error annotation taxon- omy would not only help in comparing different NLP system outputs for performance analysis, but it would also aid in developing automatic or semi- automatic error metrics for various NLP tasks (van Miltenburg et al., 2021). In this paper, we explore to what extent a standard has evolved in current error annotation schemes, and whether or not enough consensus is present to turn into a standardised consensus taxon- omy for errors of content/meaning. Our exploration has resulted in the following contributions: 1. A systematic survey of error annotation schemes comprising content/meaning error types (Section 3 and Table 1); 2. A collated list of all content/meaning error type definitions found in the papers in the sur- vey (see Appendix); 3. The minimally merged taxonomy comprising all non-task and non-domain-specific error types from the above list (Section 5.1 and Fig- ure 1); 4. A standardised and …

Content truncated to 500 words