Computer-assisted Speaker Diarization: How to Evaluate Human Corrections

Computer-assisted Speaker Diarization: How to Evaluate Human Corrections Broux Pierre-Alexandre1,2, Doukhan David2, Petitrenaud Simon1, Meignier Sylvain1, Carrive Jean2 1: Computer Science Laboratory Le Mans University (LIUM - EA 4023), Le Mans, France 2: French National Audiovisual Institute (INA), Paris, France pabroux@ina.fr, ddoukhan@ina.fr, simon.petit-renaud@univ-lemans.fr, sylvain.meignier@univ-lemans.fr, jcarrive@ina.fr Abstract In this paper, we present a framework to evaluate the human corrections of a speaker diarization. We propose four elementary actions to correct the diarization and an automaton to simulate the correction sequence. A metric is described to evaluate the correction cost. The framework is evaluated using French broadcast news drawn from the REPERE corpus. Keywords: Speaker diarization, annotation, Human-Computer Interaction (HCI), evaluation 1. Introduction The work presented in this paper has been realized to cope with some needs of the French National Audiovisual In- stitute (INA). INA is a public institution in charge of the preservation and the promotion of French audiovisual her- itage. The promotion task partly relies on the annotation of audiovisual document collections. The annotation consists in enriching the documents with summaries, keywords or participant names in order to satisfy the complex queries elaborated by INA customers or researchers within media databases. However, due to the increasing number of documents and the limited number of annotators, many documents remain undocumented or only partly documented. The informa- tion provided by the annotation greatly varies according to the kind of archives: the broadcast news is usually finely annotated, while the other programs such as games, docu- mentaries, variety shows or reality shows are much less an- notated. Thus, enterprises owning large undocumented or partly documented collections such as INA need to exploit their resources even better. One of the solutions to facilitate the annotation and improve the access to its documents is to use automatic speech and speaker recognition technolo- gies as proposed in Charhad et al. (2005; Ordelman et al. (2009; Vallet et al. (2016). The speaker diarization task is a necessary pre-processing step for speaker identification (Bonastre et al., 2000) or speech transcription (Anguera et al., 2012) in broadcast shows. The speaker diarization and speaker identification tasks allow to determine « who spoke when ». Speaker di- arization systems are generally based on unsupervised seg- mentation and clustering methods, in charge of estimating the number of speakers, and splitting the audio stream into labeled speech segments assigned to anonymous speakers. However, state-of-the-art speaker diarization systems are still not sufficiently accurate to be employed into most of INA’s applications, mainly because of the wide variety of INA’s collections. The variety relates to the time period (from the end of the nineteenth century to nowadays), the type of broadcast or the recording conditions. For these various reasons, human interventions are most of the time required to obtain robust annotations. Entirely manual annotation of speech cannot be a reason- able solution as it is a very expensive process. Indeed, nine hours are required to perform the manual annotation corre- sponding to one hour of spontaneous speech (speech tran- scription and speaker identity) (Bazillon …

Content truncated to 500 words