Towards Speaker Verification for Crowdsourced Speech Collections

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5929–5937 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5929 Towards Speaker Verification for Crowdsourced Speech Collections John Mendonc¸a1,2,*, Rui Correia3, Mariana Lourenc¸o3, Jo˜ao Freitas3, Isabel Trancoso1,2 1INESC-ID, Lisbon, Portugal 2Instituto Superior T´ecnico (IST), University of Lisbon, Portugal 3Defined.AI, Lisbon {john.mendonca, isabel.trancoso}@tecnico.ulisboa.pt, {correia, mariana.lourenco, joao}@defined.ai Abstract Crowdsourcing the collection of speech provides a scalable setting to access a customisable demographic according to each dataset’s needs. The correctness of speaker metadata is especially relevant for speaker-centred collections - ones that require the collection of a fixed amount of data per speaker. This paper identifies two different types of misalignment present in these collections: Multiple Accounts misalignment (different contributors map to the same speaker), and Multiple Speakers misalignment (multiple speakers map to the same contributor). Based on state-of-the-art approaches to Speaker Verification, this paper proposes an unsupervised method for measuring speaker metadata plausibility of a collection, i.e., evaluating the match (or lack thereof) between contributors and speakers. The solution presented is composed of an embedding extractor and a clustering module. Results indicate high precision in automatically classifying contributor alignment (> 0.94). Keywords: Crowdsourcing, Speaker Verification, Datasets 1. Introduction The success of Deep Neural Network-based solutions, which increasingly achieve (and surpass) human-level performance in several tasks, have allowed for Artifi- cial Intelligence (AI) systems to incorporate our quo- tidian life in a multitude of areas, ranging from shop- ping, to banking, social media, or security, to name a few (Grace et al., 2018). The ubiquitous adoption of AI, paired with said systems’ intrinsic characteristics, have put a high pressure on the sourcing of training data by Machine Learning practitioners (Halevy et al., 2009). Crowdsourcing has been establishing itself as an al- ternative data sourcing paradigm, producing data with quality standards that are comparable to that of ex- perts (Behrend et al., 2011). However, it does impose a new set of challenges. To get the most out of the “wisdom of the crowds” (Surowiecki, 2005) several methods to detect low quality work have been devel- oped, including follow-up validation tasks, gold stan- dard comparison (Snow et al., 2008), agreement be- tween contributors (Aroyo and Welty, 2015), and be- havioural capturing techniques (Rzeszotarski and Kit- tur, 2011). But while these strategies address the quality of the data itself, they do not tackle the correctness of the con- tributors’ self-reported metadata, typically provided during sign-up, and that, for a variety of reasons (in- cluding mistakes) may not correspond to the actual pro- file of the individual. Contributor metadata, as dis- cussed above, is of importance to mitigate biases in the data, and can include dimensions such as age, gender, country of origin, languages spoken and corresponding proficiency levels. For the particular case of speech data collections, * J.M. completed most of this work at Defined.AI. which is the main subject of the present study, contrib- utors’ metadata is a critical component, as it is not only used to …

Content truncated to 500 words