Hollywood Identity Bias Dataset: A Context Oriented Bias Analysis of Movie Dialogues

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5274–5285 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5274 Hollywood Identity Bias Dataset: A Context Oriented Bias Analysis of Movie Dialogues Sandhya Singh∗, Prapti Roy∗, Nihar Sahoo∗, Niteesh Mallela∗, Himanshu Gupta∗, Pushpak Bhattacharyya∗, Milind Savagaonkar†, Nidhi†, Roshni Ramnani†, Anutosh Maitra†, Shubhashis Sengupta† ∗CFILT, Indian Institute of Technology Bombay, India †Accenture Labs, India {sandhyasingh, nihar, niteesh, pb}@cse.iitb.ac.in, gupta himanshu@iitb.ac.in, praptir1994@gmail.com, nidhidr18@gmail.com {milind.savagaonkar, roshni.r.ramnani, anutosh.maitra, shubhashis.sengupta}@accenture.com, Abstract Warning: This paper contains content that may be offensive or upsetting however this cannot be avoided owing to the nature of the work. Movies reflect society and also hold power to transform opinions. Social biases and stereotypes present in movies can cause extensive damage due to their reach. These biases are not always found to be the need of storyline but can creep in as the author’s bias. Movie production houses would prefer to ascertain that the bias present in a script is the story’s demand. Today, when deep learning models can give human-level accuracy in multiple tasks, having an AI solution to identify the biases present in the script at the writing stage can help them avoid the inconvenience of stalled release, lawsuits, etc. Since AI solutions are data intensive and there exists no domain specific data to address the problem of biases in scripts, we introduce a new dataset of movie scripts that are annotated for identity bias. The dataset contains dialogue turns annotated for (i) bias labels for seven categories, viz., gender, race/ethnicity, religion, age, occupation, LGBTQ, and other, which contains biases like body shaming, personality bias, etc. (ii) labels for sensitivity, stereotype, sentiment, emotion, emotion intensity, (iii) all labels annotated with context awareness, (iv) target groups and reason for bias labels and (v) expert-driven group-validation process for high quality annotations. We also report various baseline performances for bias identification and category detection on our dataset. Keywords: Movie Bias, Social Bias, Hollywood Movies 1. Introduction Movie is an audio-visual form of art dealing with the complex human psyche. It nourishes and nurtures the social ethos and delves deep into the reality of society. Stemming from the social fabric, they have the poten- tial to influence a predominantly large section of public opinion. Social identity has shaped one’s life and thus, has emerged as vital for survival. Thus, social biases and stereotypes in movies can lead to significant nega- tive impacts. Identifying bias in natural language pro- cessing (NLP) tasks have seen a spurt in recent times. Bias brings along the notion of harm to the task in con- sideration (Bishop and Nasrabadi, 2006; Caliskan et al., 2017) . In contemporary times, the presence of bi- ases acts as a major deterrent toward the path of equal- ity. In this work, we define bias as a prejudice in favor or against a person, group, or community directed to- wards their social identity. Biases in movies emerge not only as the demand of sto- ryline …

Content truncated to 500 words