Curating a Large-Scale Motivational Interviewing Dataset Using Peer Support Forums

Proceedings of the 29th International Conference on Computational Linguistics, pages 3315–3330 October 12–17, 2022. 3315 Curating a Large-Scale Motivational Interviewing Dataset using Peer Support Forums Anuradha Welivita and Pearl Pu School of Computer and Communication Sciences École Polytechnique Fédérale de Lausanne Switzerland {kalpani.welivita,pearl.pu}@epfl.ch Abstract A significant limitation in developing therapeu- tic chatbots to support people going through psychological distress is the lack of high- quality, large-scale datasets capturing conver- sations between clients and trained counselors. As a remedy, researchers have focused their attention on scraping conversational data from peer support platforms such as Reddit. But the extent to which the responses from peers align with responses from trained counselors is un- derstudied. We address this gap by analyzing the differences between responses from coun- selors and peers by getting trained counselors to annotate ≈17K such responses using Motiva- tional Interviewing Treatment Integrity (MITI) code, a well-established behavioral coding sys- tem that differentiates between favorable and unfavorable responses. We developed an an- notation pipeline with several stages of quality control. Due to its design, this method was able to achieve 97% of coverage, meaning that out of the 17.3K responses we successfully labeled 16.8K with a moderate agreement. We use this data to conclude the extent to which conversa- tional data from peer support platforms align with real therapeutic conversations and discuss in what ways they can be exploited to train therapeutic chatbots. 1 Introduction Demands of the modern world are increasingly re- sponsible for bringing adverse impacts on our men- tal health. World Health organization estimates psychological distress affects 29% of people in their lifetime (Steel et al., 2014). Shortage of men- tal health workers and the stigma associated with mental health further demotivates people in actively seeking out help. Thus, provision of mental health support through the use of AI-driven conversational agents to complement traditional therapy has be- come an interesting area of research (Fitzpatrick et al., 2017; Inkster et al., 2018; Mousavi et al., 2021). But one challenge associated with develop- ing such agents is the lack of large-scale psycho- therapeutic conversations. They are either limited or are not available publicly due to ethical reasons. Nowadays, with the expansion of social media, it could be observed that people use social media platforms such as Reddit to vent their distress and peers are seen to actively respond to such posts. These conversations are available in abundance and are publicly accessible through web scraping APIs. Thus, conversations scraped from such platforms are seen as an alternative to overcome the above challenge (Welivita and Pu, 2022). Prior work has found that responses from peers contain higher em- pathic concern for posts for seeking help as many peers share similar distressful experiences (Hodges et al., 2010). But the extent to which responses from peers align with responses from trained coun- selors remain a major limitation. Knowing these differences can shed light on to what extent such conversational data could be used in training thera- peutic chatbots and what pre-processing or rephras- ing …

Content truncated to 500 words