UAntwerp at SemEval-2021 Task 5: Spans are Spans, stacking a binary word level approach to toxic span detection

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 898–903 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 898 UAntwerp at SemEval-2021 Task 5: Spans are Spans, stacking a binary word level approach to toxic span detection Ben Burtenshaw and Mike Kestemont Antwerp Centre for Digital Humanities and Literary Criticism University of Antwerp Prinsstraat 13, 2000 Antwerp (Belgium) firstname.lastname@uantwerpen.be Abstract This paper describes the system developed by the Antwerp Centre for Digital humanities and literary Criticism [UAntwerp] for toxic span detection. We used a stacked generalisation ensemble of five component models, with two distinct interpretations of the task. Two mod- els attempted to predict binary word toxicity based on ngram sequences, whilst 3 categori- cal span based models were trained to predict toxic token labels based on complete sequence tokens. The five models’ predictions were en- sembled within an LSTM model. As well as describing the system, we perform error analy- sis to explore model performance in relation to textual features. The system described in this paper scored 0.6755 and ranked 26th. 1 Introduction SemEval 2021 Task 5: Toxic Spans Detection was organised by John Pavlopoulos and colleagues, and described in detail in their task description paper (Pavlopoulos et al., 2021). Competing teams were asked to develop systems capable of detecting spans of toxic text. Predictions were evaluated using a pairwise F1-score of toxic character offset predictions, described in section 5.1. Initial analysis of the development data revealed that toxic spans were varied in content and not limited to single words. Though most examples contained single toxic words or phrases, others contained longer spans and complete sentences. Figure 1 illustrates this phenomena. With this in mind, we sort a strategy that combined longer span based detection with binary word classification. Table 1 reveals that toxic spans were on average ˜3 times longer in the development set, whilst stop words were ˜4 times more frequent. Figures 8 and 9 shows the frequency of these features in relation to model performance. # 38 ... while brutal scum hold sway in our institu- tions. # 36 Why even bother to give this woman a time of day. Bury her in the cotton field where she belong. # 31 other religions aren’t carrying out the violent and extremist doctrines as Islamists are. # 34 I hope he and other car theves read this, I’ll shoot you if I catch you in my vehicles stealing then call the police. Figure 1: Example of possible annotation rationales. dev test mean total std mean total std TOX words 3.22 25547 9.37 1.24 2488 3.59 stop ws 0.98 7802 3.82 0.26 529 1.73 NOT words 32.52 258165 34.39 31.44 62884 30.87 stop ws 16.19 128533 17.55 15.45 30909 15.75 support 7939 2000 Table 1: Macro values for toxic and non-toxic spans for development and test data. Mean average, total number, and standard deviation are shown for all words and stop words. Support is shown from development and test data as the total number …

Content truncated to 500 words