Reproducibility and Automation of the Appraisal Taxonomy

Proceedings of the 29th International Conference on Computational Linguistics, pages 3731–3740 October 12–17, 2022. 3731 Reproducibility and automation of the APPRAISAL taxonomy Pradeesh Parameswaran, Andrew Trotman, Veronica Liesaputra, David Eyers Department of Computer Science University of Otago New Zealand [pradeesh,andrew,veronica,dme]@cs.otago.ac.nz Abstract There is a lack of reproducibility in the re- sults of experiments that apply the APPRAISAL taxonomy. APPRAISAL is widely used by lin- guists to study how people judge things or peo- ple. Automating APPRAISAL could be benefi- cial for use cases such as moderating online comments. Past work on APPRAISAL anno- tation has been descriptive in nature, and the lack of publicly available data sets hinders the progress of automation. In this work, we are interested in two things; first, we are interested in how well humans can reproduce the anno- tation of APPRAISAL of the Australasian Lan- guage Technology Association (ALTA) data set. We employed four annotators, each with a similar cultural and linguistics background to reannotate the data set. Second, we are in- terested in measuring the performance of the existing automated approaches to APPRAISAL classification. Our results show a poor level of agreement at more detailed APPRAISAL cat- egories (Fleiss κ = 0.059) and a fair level of agreement (κ = 0.372) at coarse-level cate- gories. We find similar results when using automated approaches that are publicly avail- able. Our empirical evidence suggests that, at present, automating APPRAISAL classification is practical only when considering coarse-level categories of the taxonomy. 1 Introduction With the rising popularity of social media plat- forms, such as Twitter and Facebook, we are ex- periencing an unprecedented surge of unstructured textual discourse ready to be analysed (Gundecha and Liu, 2012). Supervised learning in Natural Lan- guage Processing (NLP) has helped us extract the richness of the information found in these texts for purposes such as sentiment analysis (Zhang et al., 2018), hate-speech detection (Schmidt and Wie- gand, 2019), and question answering (Shah et al., 2019). The training task within supervised learn- ing requires high-quality annotated data in order to perform well (Ramas et al., 2021). Supervised learning can apply theories of evalu- ative language (Bateman et al., 2019). With high- quality annotated data, we are confident that the task of identifying phrases of evaluative language can be automated. Evaluative language allows us to analyse how we express our feelings, our assess- ments of people, situations and objects (Benamara et al., 2017). As evaluative language is such a large and intricate discipline, herein we restrict ourselves to just the APPRAISAL1(Martin and White, 2003) taxonomy within it. APPRAISAL gives linguists a systematic ap- proach to evaluating language such as identifying and understanding how people make judgements about things (people and objects). The taxonomy has been widely used by linguists to analyse the language choices and attitudes used by writers in order to express their stances (Chen, 2022) in vari- ous media such as in news biographies, examiners’ reports and tweets (Starfield et al., 2015; Ross and Caldwell, 2020; Su and Hunston, 2019). Looking …

Content truncated to 500 words