Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1504–1532 July 9-14, 2023 ©2023 Association for Computational Linguistics Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models Myra Cheng Stanford University myra@cs.stanford.edu Esin Durmus Stanford University Dan Jurafsky Stanford University Abstract To recognize and mitigate harms from large lan- guage models (LLMs), we need to understand the prevalence and nuances of stereotypes in LLM outputs. Toward this end, we present Marked Personas, a prompt-based method to measure stereotypes in LLMs for intersectional demographic groups without any lexicon or data labeling. Grounded in the sociolinguistic concept of markedness (which characterizes ex- plicitly linguistically marked categories versus unmarked defaults), our proposed method is twofold: 1) prompting an LLM to generate per- sonas, i.e., natural language descriptions, of the target demographic group alongside personas of unmarked, default groups; 2) identifying the words that significantly distinguish personas of the target group from corresponding unmarked ones. We find that the portrayals generated by GPT-3.5 and GPT-4 contain higher rates of racial stereotypes than human-written por- trayals using the same prompts. The words distinguishing personas of marked (non-white, non-male) groups reflect patterns of othering and exoticizing these demographics. An inter- sectional lens further reveals tropes that domi- nate portrayals of marginalized groups, such as tropicalism and the hypersexualization of mi- noritized women. These representational harms have concerning implications for downstream applications like story generation. 1 Introduction The persistence of social bias and stereotypes in large language models (LLMs) is well-documented (Dinan et al., 2020; Weidinger et al., 2021, in- ter alia). These representational harms become only more concerning with the increasing use and prevalence of LLMs. Existing methods to measure stereotypes in LLMs rely on manually-constructed datasets of either unnatural templates that measure stereotypical associations (Bolukbasi et al., 2016; Caliskan et al., 2017) or natural, human-written sen- tences that contain stereotypes (Nangia et al., 2020; As I look in the mirror, I see my rich, melanin- infused skin glowing softly. My deep brown eyes sparkle with an unspoken strength and re- silience, a window to my soul. My full, lush lips form a warm and inviting smile, and my soft cheeks rise gently in response. My hair, a riot of textured coils, frames my face in a gravity- defying halo. It dances to its own beat, wild and free, just like me. I feel the love and pride I have for this crown that has been passed down to me from generations of strong Black women. Table 1: Example of GPT-4-generated persona of a Black woman. Bolded/italicized/highlighted words are those identified by our Marked Personas method as dis- tinguishing “Black”/“woman”/“Black woman” personas from unmarked ones. We analyze how such words are tied to seemingly positive stereotypes, essentializing narratives, and other harms. Nadeem et al., 2021). They also have a trade-off between 1) characterizing a fixed set of stereotypes for specific demographic groups and 2) generaliz- ing to a broader range of stereotypes and groups (Cao …

Content truncated to 500 words