ArMIS - The Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2282–2291 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2282 ArMIS - The Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements Dina Almanea and Massimo Poesio Queen Mary University of London {d.almanea, m.poesio}@qmul.ac.uk Abstract The use of misogynistic and sexist language has increased in recent years in social media, and is increasing in the Arabic world in reaction to reforms attempting to remove restrictions on women lives. However, there are few benchmarks for Arabic misogyny and sexism detection, and in those the annotations are in aggregated form even though misogyny and sexism judgments are found to be highly subjective. In this paper we introduce an Arabic misogyny and sexism dataset (ArMIS) characterized by providing annotations from annotators with different degree of religious beliefs, and provide evidence that such differences do result in disagreements. To the best of our knowledge, this is the first dataset to study in detail the effect of beliefs on misogyny and sexism annotation. We also discuss proof-of-concept experiments showing that a dataset in which disagreements have not been reconciled can be used to train state-of-the-art models for misogyny and sexism detection; and consider different ways in which such models could be evaluated. Keywords: misogyny, sexism, subjectivity, disagreement 1. Introduction The growth in recent years of social networks and online discussions groups has unfortunately been ex- ploited by some individuals to spread offensive views about people with certain characteristics (religion, eth- nicity, women, etc). Although the offenders are not a majority, these offensive views can affect and stigma- tize individuals such as feminists attacked in misogy- nous and sexist posts. This phenomenon has motivated much research on detecting offensive and hate texts on- line. Studies have found that women are especially subject to offensive language on social media (Frenda et al., 2019). (According to Amnesty International, an of- fending remark towards a woman politician or journal- ist appears on social media every 30 seconds.1) Our work is concerned with detecting misogynistic and sex- ist tweets, focusing in particular on Arabic. The spread of this type of offensive language has recently increased in Saudi Arabia due to the changes brought about by the Saudi government’s Vision 2030 project aiming to re- lax restrictions on women activities2, for example laws allowing women to drive and/or to be employed in high- level government positions. These changes have orig- inated a major debate in Saudi Arabia: some people agree with these reforms, but others do not. Unfortu- nately such debates often involve the use of offensive language or even hate speech. The research in this paper is motivated by a feature of work on misogynistic and sexist language detection that has not attracted much attention so far–namely, the fact 1https://www.amnesty.org.uk/press-releases/women- abused-twitter-every-30-seconds-new-study 2https://english.alarabiya.net/features/2020/10/18/Top- 10-moments-for-Saudi-Arabian-women-since-Vision-2030 that judgments about what counts as offensive appears to very much depend on certain characteristics of those making the judgments. Women appear to find different texts misogynistic than men, …

Content truncated to 500 words