Logic-driven Indirect Supervision: An Application to Crisis Counseling

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11704–11722 July 9-14, 2023 ©2023 Association for Computational Linguistics Logic-driven Indirect Supervision: An Application to Crisis Counseling Mattia Medina Grespan1, Meghan Broadbent2, Xinyao Zhang2, Katherine E. Axford2, Brent Kious3, Zac Imel2 and Vivek Srikumar1 1Kahlert School of Computing, University of Utah 2Department of Educational Psychology, University of Utah 3Huntsman Mental Health Institute, Department of Psychiatry, University of Utah mattiamg@cs.utah.edu Abstract Ensuring the effectiveness of text-based cri- sis counseling requires observing ongoing conversations and providing feedback, both labor-intensive tasks. Automatic analysis of conversations—at the full chat and utterance levels—may help support counselors and pro- vide better care. While some session-level train- ing data (e.g., rating of patient risk) is often available from counselors, labeling utterances requires expensive post hoc annotation. But the latter can not only provide insights about conversation dynamics, but can also serve to support quality assurance efforts for counselors. In this paper, we examine if inexpensive—and potentially noisy—session-level annotation can help improve label utterances. To this end, we propose a logic-based indirect supervision ap- proach that exploits declaratively stated struc- tural dependencies between both levels of anno- tation to improve utterance modeling. We show that adding these rules gives an improvement of 3.5% f-score over a strong multi-task baseline for utterance-level predictions. We demonstrate via ablation studies how indirect supervision via logic rules also improves the consistency and robustness of the system. Trigger warning: This paper discusses suicide in the context of crisis counseling and includes exam- ples illustrating such conversations. 1 Introduction Text-based crisis counseling services like Crisis Text Line1 and the 988 Suicide & Crisis Lifeline2 are increasingly adopted by people seeking con- fidential mental health support. They help thou- sands of texters every day. But the volume of users challenges the ability of crisis systems to provide consistently high-quality service. For example, in our experience with the regional suicide hotline, 1https://www.crisistextline.org 2https://988lifeline.org/ SafeUT,3 we found that counselors can have ex- tended shifts involving up to eight conversations with potentially suicidal clients simultaneously! Figure 1a shows an illustrative anonymized exam- ple of such a session. Addressing the twin problems of managing coun- selor workload and ensuring quality requires train- ing new counselors and providing feedback to ex- isting ones. In particular, understanding suicide risk in client utterances may help counselors learn to prioritize high-risk client situations, especially when dealing with multiple chats simultaneously or when fatigued. As Imel et al. (2017) note, scaling such efforts requires technological assistance. Pre- vious work (e.g., Broadbent et al., 2023; Guzman- Nateras et al., 2022; Shrestha et al., 2021; Haque et al., 2020) has shown that NLP models can reli- ably assess risk in crisis chats. Yet, building models for risk assessment at the utterance level is chal- lenging because of the dearth of training data. Utterance-level risk labeling requires post hoc annotation by experts who follow a coding manual; the process can be slow and expensive. In contrast, session-level risk data is …

Content truncated to 500 words