Have a Cake and Eat it Too: Assessing Discriminating Performance of an Intelligibility Index Obtained from a Reduced Sample Size

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1791–1795 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1791 Have a Cake and Eat it Too: Assessing Discrimination Performance of an Intelligibility Index Obtained from a Reduced Sample Size A. Marczyk1, A. Ghio1, M. Lalain1, M. Rebourg1, C. Fredouille2, V. Woisard3, 4 1Aix-Marseille Université CNRS-LPL, Aix-en-Provence, France; 2 LIA, Avignon University, France; 3Service ORL, CHU Larrey; 4URI Octogone-Lordat, Toulouse, France {anna.marczyk. alain.ghio. muriel.lalain. marie.rebourg}@lpl-aix.fr, corinne.fredouille@univ-avignon.fr, woisard.v@chu-toulouse.fr Abstract This paper investigates random vs. phonetically motivated reduction of linguistic material used in an intelligibility task in speech disordered populations and the subsequent impact on the discrimination classifier quantified by the area under the receiver operating characteristics curve (AUC of ROC). The comparison of obtained accuracy indexes shows that when the sample size is reduced based on a phonetic criterium—here, related to phonotactic complexity—, the classifier has a higher ranking ability than when the linguistic material is arbitrarily reduced. Crucially, downsizing the linguistic sample to about 30% of the original dataset does not diminish the discriminatory performance of the classifier. This result is of significant interest to both clinicians and patients as it validates a tool that is both reliable and efficient. Keywords: speech intelligibility index, discrimination ability, speech disorders, ROC curve 1. Introduction Speech intelligibility measure1 represents a scalar metric that reflects how well a message is decoded by a listener and gives a reliable estimate of word-form impairments, a cardinal symptom of common speech disorders such a dysarthria. It serves multiple purposes in clinical settings, from being used as a diagnostic tool for patient discrimination and for determining the degree of impairment, to serving as an intervention monitoring indicator. It has been typically obtained based on linguistically controlled single word and sentence material read by patients and orthographically transcribed by independent listeners (Kent, Weismer, Kent, & Rosenbek, 1989). Since interferences due to word familiarity are an important caveat of using word material for clinical speech assessment, intelligibility metrics have been more recently obtained from pseudoword lists (e.g. Allen, Alais, & Carlile, 2012). A phonetically controlled pseudoword material designed for French-speaking speech disordered population has been recently proposed for the acoustic- phonetic decoding task with the aim of circumventing confounds due to the lexical bias, such as phonemic restauration or learning effects (Perceived Phonological Deviation score, henceforth PPD, Ghio et al., 2016, 2018; Lalain et al., under review). Next to the type of linguistic material used for intelligibility assessment is the issue related to the trade-off between the index ranking ability and the volume of data needed to generate statistically reliable results. Clinical practice imposes considerable time constraints on practitioners and obtaining a reliable and efficient speech performance classifier becomes critical. While PPD intelligibility score proved to be highly performant in discriminating between healthy and carcinologic speakers (Ghio et al., 2018), it was generated from lists of 52 pseudowords, a sample that requires an important amount of time from a clinician’s perspective. For …

Content truncated to 500 words