Do self-supervised speech models develop human-like perception biases?

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7591 - 7605 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Do self-supervised speech models develop human-like perception biases? Juliette Millet CoML, ENS/CNRS/EHESS/INRIA/PSL, LLF, University of Paris, CNRS, CRI, FAN, IIFR, University of Paris, Paris, France juliette.millet@cri-paris.org Ewan Dunbar CoML, ENS/CNRS/EHESS/INRIA/PSL, Paris, France University of Toronto, Toronto, Canada ewan.dunbar@utoronto.ca Abstract Self-supervised models for speech process- ing form representational spaces without us- ing any external labels. Increasingly, they ap- pear to be a feasible way of at least partially eliminating costly manual annotations, a prob- lem of particular concern for low-resource lan- guages. But what kind of representational spaces do these models construct? Human per- ception specializes to the sounds of listeners’ native languages. Does the same thing happen in self-supervised models? We examine the representational spaces of three kinds of state- of-the-art self-supervised models: wav2vec 2.0, HuBERT and contrastive predictive cod- ing (CPC), and compare them with the percep- tual spaces of French-speaking and English- speaking human listeners, both globally and taking account of the behavioural differences between the two language groups. We show that the CPC model shows a small native lan- guage effect, but that wav2vec 2.0 and Hu- BERT seem to develop a universal speech per- ception space which is not language specific. A comparison against the predictions of super- vised phone recognisers suggests that all three self-supervised models capture relatively fine- grained perceptual phenomena, while super- vised models are better at capturing coarser, phone-level, effects of listeners’ native lan- guage, on perception. 1 Introduction Recent advances in speech recognition and repre- sentation learning show that self-supervised pre- training is an excellent way of improving perfor- mance while reducing the amount of labelled data needed for training. For example, for the Lib- riSpeech dataset (Panayotov et al., 2015), the cur- rent best word error rates (Xu et al., 2021; Zhang et al., 2020) are obtained by systems based on the self-supervised wav2vec 2.0 model (Baevski et al., 2020). Systems using self-supervised pre-training, both using wav2vec 2.0 and using HuBERT (Hsu et al., 2021a,b), show excellent word error rates after having been fine-tuned on only ten minutes of labelled data. What is the effect of this self-supervised pre- training? What type of representational spaces are learned by these models? Lakhotia et al. (2021) compared wav2vec 2.0, HuBERT, and contrastive predictive coding (CPC: Oord et al. 2017; Rivière and Dupoux 2021) using an ABX discriminability metric (Schatz, 2016), demonstrating that all three models preserve and enhance linguistically rele- vant speech sound contrasts in the language they are trained on. We build on this work, asking how these representational spaces compare to the per- ceptual spaces of human listeners, as inferred from behaviour on phone discrimination experiments. Human listeners develop speech perception bi- ases under the influence of their native languages. For example, Japanese native speakers tend to con- fuse the English sounds /r/ and /l/ (Yamada and Tohkura, 1990) (right and light …

Content truncated to 500 words