Are Experts Needed? On Human Evaluation of Counselling Reflection Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6906–6930 July 9-14, 2023 ©2023 Association for Computational Linguistics Are Experts Needed? On Human Evaluation of Counselling Reflection Generation Zixiu Wu12 Simone Balloccu34 Ehud Reiter3 Rim Helaoui1 Diego Reforgiato Recupero2 Daniele Riboni2 1Philips Research, the Netherlands 2University of Cagliari, Italy 3University of Aberdeen, UK 4Charles University, Czechia z.wu@studenti.unica.it {s.balloccu.19, e.reiter}@abdn.ac.uk rim.helaoui@philips.com {diego.reforgiato, riboni}@unica.it Abstract Reflection is a crucial counselling skill where the therapist conveys to the client their interpre- tation of what the client said. Language models have recently been used to generate reflections automatically, but human evaluation is chal- lenging, particularly due to the cost of hiring experts. Laypeople-based evaluation is less ex- pensive and easier to scale, but its quality is unknown for reflections. Therefore, we explore whether laypeople can be an alternative to ex- perts in evaluating a fundamental quality aspect: coherence and context-consistency. We do so by asking a group of laypeople and a group of experts to annotate both synthetic reflections and human reflections from actual therapists. We find that both laypeople and experts are re- liable annotators and that they have moderate- to-strong inter-group correlation, which shows that laypeople can be trusted for such evalua- tions. We also discover that GPT-3 mostly pro- duces coherent and consistent reflections, and we explore changes in evaluation results when the source of synthetic reflections changes to GPT-3 from the less powerful GPT-2. 1 Introduction Motivational Interviewing (MI, Miller and Roll- nick, 2012) is a highly effective counselling prac- tice in healthcare (Moyers et al., 2009), where the therapist focuses on evoking the client’s own moti- vation for behaviour change, such as smoking ces- sation and alcohol use reduction. In MI, reflective listening is a crucial strategy of showing empathy, where the therapist conveys a brief conversational summary of how they understand what the client said (Miller et al., 2003; Rollnick et al., 2008). An example is shown in Table 1. Learning effective reflective listening requires considerable training time and expert supervi- sion (Rautalinko and Lisper, 2004; Rautalinko et al., 2007). Therefore, recent studies used lan- guage models (LMs) as automatic reflection genera- tors to aid training (Shen et al., 2020, 2022; Ahmed, Context Client: Well, I’m here because my mom wants me to be here. Therapist: Mm-hmm. Client: I don’t really wanna be here, but it-it– whatever. Therapist: Got it. Client: Um, she-she found my stash- Therapist: Uh-huh. Client: -and she freaked out, and she’s going crazy over it. Um, I don’t why she was going through stuff in the first place, but whatever, so, now I’m here. · · · (intermediate turns) Therapist: And, uh, sounds like you’re-you’re pretty upset with your mom for-for doing that? Client: I am. Therapist: Yeah. Client: I mean, it’s my stuff, I don’t know why she’s– Reflection Candidates Therapist (Human): Right. It’s like your private place and, you know, it’s– that’s– it’s your stuff. GPT-2: It’s a very sad thing. GPT-3: It sounds …

Content truncated to 500 words