Speaker Overlap-aware Neural Diarization for Multi-party Meeting Analysis

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7458 - 7469 December 7-11, 2022 ©2022 Association for Computational Linguistics Speaker Overlap-aware Neural Diarization for Multi-party Meeting Analysis Zhihao Du and Shiliang Zhang and Siqi Zheng and Zhijie Yan Speech Lab, Alibaba Group, China {neo.dzh,sly.zsl}@alibaba-inc.com Abstract Recently, hybrid systems of clustering and neural diarization models have been success- fully applied in multi-party meeting analysis. However, current models always treat over- lapped speaker diarization as a multi-label classification problem, where speaker depen- dency and overlaps are not well considered. To overcome the disadvantages, we reformu- late overlapped speaker diarization task as a single-label prediction problem via the pro- posed power set encoding (PSE). Through this formulation, speaker dependency and over- laps can be explicitly modeled. To fully leverage this formulation, we further propose the speaker overlap-aware neural diarization (SOND) model, which consists of a context- independent (CI) scorer to model global speaker discriminability, a context-dependent scorer (CD) to model local discriminability, and a speaker combining network (SCN) to combine and reassign speaker activities. Ex- perimental results show that using the pro- posed formulation can outperform the state-of- the-art methods based on target speaker voice activity detection, and the performance can be further improved with SOND, resulting in a 6.30% relative diarization error reduction. 1 Introduction Speaker identity is an import information for speaker-attributed automatic speech recognition (Carletta et al., 2005; Barker et al., 2018) and dia- logue comprehension (He et al., 2021), especially in multi-party meeting scenarios (Yu et al., 2022a). Recently, speaker diarization is developed to detect and track speakers with acoustic features. In gen- eral, speaker diarization methods can be divided into three groups, i.e., clustering-based algorithms, end-to-end models and hybrid systems. Clustering-based methods comprise three in- dividual parts, including speech segmentation, embedding extraction, and clustering algorithm. Specifically, a long-term audio is first split into sev- eral segments by voice activity detection (VAD). Then, speaker embeddings (Dehak et al., 2011; Wan et al., 2018; Snyder et al., 2018) are extracted for each segment, which are partitioned into clus- ters by unsupervised clustering algorithms, such as k-means (Dimitriadis and Fousek, 2017), spectral clustering (Ning et al., 2006), agglomerative hier- archical clustering (AHC) (Garcia-Romero et al., 2017), and Leiden community detection (Zheng and Suo, 2022). However, such clustering al- gorithms are in an unsupervised manner, which does not minimize the diarization errors directly. To overcome this issue, neural models are intro- duced, such as Li et al. (2021) and Zhang et al. (2022). Recently, the variational Bayesian hidden Markov model (VBx) is introduced to refine the clustering results (Díez et al., 2019). Although VBx achieves impressive performance on several datasets (Castaldo et al., 2008; Ryant et al., 2019), it has trouble handling overlapping speech due to speaker-homogeneous assumption. End-to-end (E2E) approaches treat speaker di- arization as a multi-label classification problem, where a deep neural network is employed to predict a set of binary labels (speaker activities) for each timestep. In this way, E2E models can deal with …

Content truncated to 500 words