Evaluating the Performance of Transformer-based Language Models for Neuroatypical Language

Proceedings of the 29th International Conference on Computational Linguistics, pages 3412–3419 October 12–17, 2022. 3412 Evaluating the Performance of Transformer-based Language Models for Neuroatypical Language Duanchen Liu1,2, Zoey Liu1, Qingyun Yang1,3, Yujing Huang1, Emily Prud’hommeaux1 1Department of Computer Science, Boston College, Chestnut Hill MA, USA 2MIT, Cambridge MA, USA 3Cornell Tech, New York NY, USA liudc@mit.edu, {liuaal,huangac,prudhome}@bc.edu, qny2@cornell.edu Abstract Difficulties with social aspects of language are among the hallmarks of autism spectrum disor- der (ASD). These communication differences are thought to contribute to the challenges that adults with ASD experience when seeking em- ployment, underscoring the need for interven- tions that focus on improving areas of weakness in pragmatic and social language. In this paper, we describe a transformer-based framework for identifying linguistic features associated with social aspects of communication using a corpus of conversations between adults with and with- out ASD and neurotypical conversational part- ners produced while engaging in collaborative tasks. While our framework yields strong accu- racy overall, performance is significantly worse for the language of participants with ASD, sug- gesting that they use a more diverse set of strate- gies for some social linguistic functions. These results, while showing promise for the devel- opment of automated language analysis tools to support targeted language interventions for ASD, also reveal weaknesses in the ability of large contextualized language models to model neuroatypical language. 1 Introduction Autism spectrum disorder (ASD) is a neurodevel- opmental disorder with an estimated global preva- lence of 1 in 100 worldwide (Zeidan et al., 2022). The majority of people diagnosed with ASD today are verbal (Rose et al., 2016) and have average or above average intellectual ability (Christensen et al., 2016). Nevertheless, young adults with ASD are employed at significantly lower rates than their peers with other neurodevelopmental conditions, including learning disabilities and intellectual dis- ability (Shattuck et al., 2012). Difficulty with social communication is one of the diagnostic criteria for ASD (American Psychi- atric Association, 2013) and is reported to be one of the strongest contributors to negative professional outcomes (Hurlbutt and Chalmers, 2004; Baldwin et al., 2014). For this reason, language skills are commonly targeted for intervention in individu- als with ASD (Parsons et al., 2017), but it can be difficult to identify specific areas in need of reme- diation. Most prior work on quantifying pragmatic deficits in ASD has relied on manual analysis of speech transcripts (Loukusa et al., 2007; Paul et al., 2009; Conlon et al., 2019), a process that is time consuming and requires expertise. In this paper, we use a spoken language corpus collected specifically for training automatic sys- tems to explore social communication and pragmat- ics in ASD. The corpus consists of transcribed con- versations between adults with and without ASD as they engage with neurotypical interlocutors in collaborative tasks designed to resemble workplace activities. We review the careful manual process of assigning pragmatic feature values and dialog act labels to each utterance. Following recent prior work on these features in other contexts, we pro- pose a …

Content truncated to 500 words