What Clued the AI Doctor In? On the Influence of Data Source and Quality for Transformer-Based Medical Self-Disclosure Detection

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1201–1216 May 2-6, 2023 ©2023 Association for Computational Linguistics What Clued the AI Doctor In? On the Influence of Data Source and Quality for Transformer-Based Medical Self-Disclosure Detection Mina Valizadeh, Xing Qian, Pardis Ranjbar-Noiey, Cornelia Caragea, and Natalie Parde Department of Computer Science University of Illinois Chicago {mvaliz2, xqian20, pranjb3, cornelia, parde}@uic.edu Abstract Recognizing medical self-disclosure is impor- tant in many healthcare contexts, but it has been under-explored by the NLP community. We conduct a three-pronged investigation of this task. We (1) manually expand and refine the only existing medical self-disclosure corpus, resulting in a new, publicly available dataset of 3,919 social media posts with clinically vali- dated labels and high compatibility with the ex- isting task-specific protocol. We also (2) study the merits of pretraining task domain and text style by comparing Transformer-based models for this task, pretrained from general, medi- cal, and social media sources. Our BERTweet condition outperforms the existing state of the art for this task by a relative F1 score increase of 16.73%. Finally, we (3) compare data aug- mentation techniques for this task, to assess the extent to which medical self-disclosure data may be further synthetically expanded. We dis- cover that this task poses many challenges for data augmentation techniques, and we provide an in-depth analysis of identified trends. 1 Introduction Self-disclosure is a complex communicative pro- cess (Kreiner and Levi-Belz, 2019) that involves sharing one’s personal thoughts, feelings, or mem- ories with another individual (Jourard and Fried- man, 1970). Reciprocal self-disclosure between conversation partners may strengthen relationships (Altman and Taylor, 1973) and improve the com- municative experience (Wang et al., 2016). Self- disclosure may take nuanced forms, such as med- ical self-disclosure (see Figure 1), or the act of disclosing symptoms, diagnoses, or other informa- tion related to mental or physical health problems (Valizadeh et al., 2021; Joinson, 2001). Some med- ical self-disclosures may be explicit (A), whereas others may be less direct (B and C). Medical self-disclosure is helpful from a clinical perspective and reinforces therapeutic relationships Figure 1: Examples of medical self-disclosure. Note that medical self-disclosure may be explicit (A) or less direct (B and C). during medical interactions (Jannat, 2018; Kadji and Mast, 2021). It also may enable earlier de- tection and treatment of medical issues (Joinson, 2001; Tidwell and Walther, 2002; Valizadeh et al., 2021). Detecting medical self-disclosure automat- ically could support clinicians and other medical practitioners in productively identifying helpful pa- tient information from untapped sources or dur- ing review of medical conversations (Farber, 2003; Stricker, 2003; Valizadeh and Parde, 2022; Kaelin et al., 2021). However, research towards automated medical self-disclosure detection has been limited and may require different techniques from those used to detect self-disclosure in the general domain (Valizadeh et al., 2021; Reuel et al., 2022). In this paper, we comprehensively investigate au- tomated medical self-disclosure detection. We man- ually expand the only existing dataset (MEDSD) in this domain to …

Content truncated to 500 words