A Comprehensive Evaluation of Incremental Speech Recognition and Diarization for Conversational AI

Proceedings of the 28th International Conference on Computational Linguistics, pages 3492–3503 Barcelona, Spain (Online), December 8-13, 2020 3492 A Comprehensive Evaluation of Incremental Speech Recognition and Diarization for Conversational AI Angus Addlesee The Interaction Lab Heriot-Watt University Edinburgh ja204@hw.ac.uk Yanchao Yu The Interaction Lab Heriot-Watt University Edinburgh y.yu@hw.ac.uk Arash Eshghi The Interaction Lab Heriot-Watt University Edinburgh a.eshghi@hw.ac.uk Abstract Automatic Speech Recognition (ASR) systems are increasingly powerful and more accurate, but also more numerous with several options existing currently as a service (e.g. Google, IBM, and Microsoft). Currently the most stringent standards for such systems are set within the context of their use in, and for, Conversational AI technology. These systems are expected to operate incrementally in real-time, be responsive, stable, and robust to the pervasive yet peculiar characteristics of conversational speech such as disfluencies and overlaps. In this paper we evaluate the most popular of such systems with metrics and experiments designed with these standards in mind. We also evaluate the speaker diarization (SD) capabilities of the same systems which will be particularly important for dialogue systems designed to handle multi-party interaction. We found that Microsoft has the leading incremental ASR system which preserves disfluent materials and IBM has the leading incremental SD system in addition to the ASR that is most robust to speech overlaps. Google strikes a balance between the two but none of these systems are yet suitable to reliably handle natural spontaneous conversations in real-time. 1 Introduction Automatic Speech Recognition (ASR) is one of the first important milestones of deep learning (Seide et al., 2011) and the standard evaluation metric for ASR systems, the Word Error Rate (WER), has been consistently improving to impressive levels (Saon et al., 2016; Saon et al., 2017; Xiong et al., 2017). But with a few notable exceptions discussed below, the suitability of these state of the art ASR systems for Conversational AI (henceforth SDS for Spoken Dialogue System1.) is rarely evaluated. To be truly natural and responsive, SDS architectures have had to be redesigned (Schlangen and Skantze, 2009; Schlangen et al., 2010; Schlangen and Skantze, 2011; Baumann and Schlangen, 2012) to take into account the inherently time-linear and incremental nature of human language processing (see Purver et al. (2009); Healey et al. (2011) & Kempson et al. (2016) among many others). These incremental architectures impose additional evaluation criteria and further constraints on the output required of ASR systems: First, ASR hypotheses should be produced incrementally, word by word, with minimal latency. This means that downstream processing (e.g. for Natural Language Understanding (NLU)) can begin before a turn is complete. This has been shown to lead to dialogue systems which feel more natural (Aist et al., 2006; Skantze and Hjalmarsson, 2010) and more satisfying to interact with (Aist et al., 2007). ASR hypotheses should furthermore be stable through time (minimising ‘jitter’ in the output) (Selfridge et al., 2011; Baumann et al., 2011; Baumann et al., 2016). Second, incremental detection and processing of disfluent speech is crucial for robust Natural Language Understanding …

Content truncated to 500 words