WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 877–892 July 9-14, 2023 ©2023 Association for Computational Linguistics How About Kind of Generating Hedges using End-to-End Neural Models? Alafate Abulimiti1,2, Chloé Clavel3, Justine Cassell1,4 1 INRIA, Paris 2 ENS/PSL <alafate.abulimiti@inria.fr> 3 LTCI, Insitut Polytechnique de Paris, Telecom Paris <chloe.clavel@telecom-paris.fr> 4 Carnegie Mellon University <justine@cs.cmu.edu> Abstract Hedging is a strategy for softening the impact of a statement in conversation. In reducing the strength of an expression, it may help to avoid embarrassment (more technically, “face threat”) to one’s listener. For this reason, it is often found in contexts of instruction, such as tutoring. In this work, we develop a model of hedge generation based on i) fine-tuning state- of-the-art language models trained on human- human tutoring data, followed by ii) reranking to select the candidate that best matches the ex- pected hedging strategy within a candidate pool using a hedge classifier. We apply this method to a natural peer-tutoring corpus containing a significant number of disfluencies, repetitions, and repairs. The results show that generation in this noisy environment is feasible with rerank- ing. By conducting an error analysis for both approaches, we reveal the challenges faced by systems attempting to accomplish both social and task-oriented goals in conversation. 1 Introduction When people interact, they attend not just to the task at hand, but also to their relationship with their interlocutors (Tracy and Coupland, 1990). One key aspect of the relationship that people attend to, while engaging in contexts as diverse as sales (Gremler and Gwinner, 2008; Planken, 2005), education (Glazier, 2016; Murphy and Rodríguez-Manzanares, 2012) and healthcare (Di- Matteo, 1979; Leach, 2005), is what is referred to as rapport, a sense of harmony and mutual un- derstanding between participants in a conversation (Spencer-Oatey, 2005; Tickle-Degnen and Rosen- thal, 1990). Indeed, higher levels of rapport are cor- related with better performance in each of these do- mains. Zhao et al. (2014) describes rapport as built upon a base of mutual attentiveness, face manage- ment, and coordination. This base is built primarily by conversational strategies, or ways of speaking (including nonverbal and paraverbal behaviors) that yeah, so it will be ... I think you could add four to both sides. oh no... Figure 1: Hedging in peer tutoring manage rapport throughout a conversation. Key conversational strategies include self-disclosure, reference to shared experience, praise, and hedging — giving instructions or conveying information in an indirect manner when it might otherwise sound rude or overly demanding. End-to-end large language models (LLM), of the kind that are increasingly popular and powerful, do a good job at carrying out the propositional or information-carrying aspects of conversation, and a relatively good job of maintaining the coherence of a conversation, but they are not as good at changing how they say something as a function of a relation- ship with the human user, while humans are, for the most part, quite good at this. However, since saying things in a specific …

Content truncated to 500 words