Automatically Predicting Judgement Dimensions of Human Behaviour

Automatically Predicting Judgement Dimensions of Human Behaviour Segun Taofeek Aroyehun CIC, Instituto Polit´ecnico Nacional Mexico City, Mexico aroyehun.segun@gmail.com Alexander Gelbukh CIC, Instituto Polit´ecnico Nacional Mexico City, Mexico www.gelbukh.com Abstract This paper describes our submission to the ALTA-2020 shared task on assessing be- haviour from short text, We evaluate the ef- fectiveness of traditional machine learning and recent transformers pre-trained models. Our submission with the Roberta-large model and prediction threshold achieved first place on the private leaderboard. 1 Introduction Language enables us to express evaluation of peo- ple, action, event, and things. This manifests as emotion and assessment of human behaviour and artefacts. The study of evaluative language has benefited from efforts in several disciplines such as linguistics, philosophy, psychology, cognitive sci- ence and computer science (Benamara et al., 2017). In linguistics, the appraisal framework of Martin and White (2003) provides a detailed classifica- tion scheme for understanding how evaluation is expressed and implied in language. In computer science, affective computing study evaluative lan- guage under the umbrella term of sentiment anal- ysis with common tasks involving detection and classification of polarity and emotion, and aspect- based sentiment analysis, among others. Sentiment analysis has benefited from the availability of user- generated content on online platforms. The theory of appraisal proposed by Martin and White (2003) has three categories of evaluative text: affect, judgement, and appreciation. These categories respectively model opinions in terms of emotions, norms, and aesthetics. Utterances are viewed as indicating positive (“praising”) or negative (“blaming”) disposition towards some ob- ject (person, thing, action, situation, or event). The judgement dimensions are normality, capac- ity, tenacity, veracity, and propriety. Each of the dimensions represents an answer to the following corresponding questions: • Normality: How special? • Tenacity: How dependable? • Capacity: How capable? • Veracity: How honest? • Propriety: How far beyond re- proach? The corpus used in this paper is annotated with the above judgement dimensions. Taboada and Grieve (2004) automatically catego- rized appraisal into affect, judgement, and apprecia- tion using a lexical approach that groups adjectives according to their semantic orientation. Benamara et al. (2017) surveyed linguistic and computational approaches to the study of evaluative text. Their analysis suggested that appraisal is a richer and more detailed task amenable to computational ap- proaches subject to availability of data. They envi- sion that appraisal analysis can contribute to the ad- vances in affective computing. Recently, Hofmann et al. (2020) showed that dimensions of appraisal can improve emotion detection in text. A similar observation was made by Whitelaw et al. (2005) who found appraisal phrases as useful features for sentiment analysis. This paper investigates the capabilities of ma- chine learning models in predicting the dimen- sions of human judgement expressed in short texts (tweets) as part of the ALTA-2020 shared task on assessing human behaviour (Moll´a, 2020). The task aims to advance computational techniques for analysing evaluative language. The use of neural networks has lead to significant performance improvements in NLP tasks. However, neural networks require a large amount of labeled …

Content truncated to 500 words