Evaluation of Automatic Speech Recognition for Conversational Speech in Dutch, English and German: What Goes Missing?

Evaluation of Automatic Speech Recognition for Conversational Speech in Dutch, English, and German: What Goes Missing? Alianda Lopez, Andreas Liesenfeld, Mark Dingemanse Centre for Language Studies Radboud University, Nijmegen, The Netherlands {ada.lopez,andreas.liesenfeld,mark.dingemanse}@ru.nl Abstract As voice user interfaces and conversational agents grow in importance, automatic speech recognition (ASR) encounters increasingly free-form and informal input data. Conversa- tional speech is at once the most challenging and the most ecologically relevant type of data for speech recognition in this context. Here we evaluate the performance of several ASR engines on conversational speech in three lan- guages, focusing on the fate of backchannels and other interactionally relevant elements of talk. We propose forms of error analysis based on ngram salience scoring that can complement default measures like word error rates (WER) and are more informative of ASR’s ability to live up to the task of accurately representing real-world interaction. 1 Introduction Conversational agents and voice-driven virtual as- sistants are becoming more and more integrated into our daily lives. However, users are still dissat- isfied with their conversational abilities, describing them as frustrating, stilted, and unnatural (Clark et al., 2019; Moore, 2017; Kopp and Krämer, 2021). One likely reason is that most automatic speech recognition (ASR) systems are trained on carefully read monological speech (Panayotov et al., 2015; Ardila et al., 2020) rather than on free-flowing in- formal conversational interaction. One of the key ways conversational speech dif- fers from read speech is the nature of its produc- tion: planned and produced in real-time by people together. Conversation bears the traces of its dialog- ical origins in the form of elements like backchan- nels (Yngve, 1970; Fujimoto, 2007), disfluencies (Ginzburg et al., 2014; Hough and Schlangen, 2017), and other forms of speech management (Allwood et al., 1990), collateral signals (Clark, 1996) and non-lexical conversational sounds (Ward, 2006). The variety of terms in this area highlights the disparate strands of research concerned with such phenomena, and also encodes an implicit eval- uation of these elements as somehow missable, marginal, or straying from the norm. Quite some work has focused on “disfluency detection”, of- ten with the goal of ‘cleaning up’ transcripts for use in downstream natural language understand- ing pipelines or for public consumption (Hough and Schlangen, 2017; Shalyminov et al., 2018; Za- yats et al., 2019). However, a recent upsurge in research shows the importance of these elements as metacommunicative tools for streamlining conver- sation (Buschmeier and Kopp, 2018; Kosmala and Morgenstern, 2018; Dingemanse and Liesenfeld, 2022), and this is where their relevance for some ASR applications lies. For instance, interjections like mhmm and uh-huh in English serve as a cue for the speaker to continue talking, while others like huh? instead indicate a need for repetition or clarification — quite an important distinction to get for voice user interfaces. Likewise, items like uh and um are easily seen as irregularities to be cleaned up, but they can also do interactional work, such as signalling upcoming complexities or inter- actionally delicate moments (Clark and Fox …

Content truncated to 500 words