Integrating Disfluency-based and Prosodic Features with Acoustics in Automatic Fluency Evaluation of Spontaneous Speech

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6429–6437 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6429 Integrating Disfluency-based and Prosodic Features with Acoustics in Automatic Fluency Evaluation of Spontaneous Speech Huaijin Deng1, Youchao Lin1, Takehito Utsuro1, Akio Kobayashi2, Hiromitsu Nishizaki3, Junichi Hoshino1 1Graduate School of Systems and Information Engineering, University of Tsukuba, Tsukuba, 305-8573, Japan 2Department of Industrial Information, Tsukuba University of Technology, Tsukuba 305-8520, Japan 3Graduate School of Interdisciplinary Research, University of Yamanashi, Kofu 400-8510, Japan {s2020815, s1730186} @ s.tsukuba.ac.jp, {utsuro, jhoshino} @ iit.tsukuba.ac.jp a-kobayashi @ a.tsukuba-tech.ac.jp, hnishi @ yamanashi.ac.jp Abstract This paper describes an automatic fluency evaluation of spontaneous speech. In the task of automatic fluency evaluation, we integrate diverse features of acoustics, prosody, and disfluency-based ones. Then, we attempt to reveal the contribution of each of those diverse features to the task of automatic fluency evaluation. Although a variety of different disfluencies are observed regularly in spontaneous speech, we focus on two types of phenomena, i.e., filled pauses and word fragments. The experimental results demonstrate that the disfluency-based features derived from word fragments and filled pauses are effective relative to evaluating fluent/disfluent speech, especially when combined with prosodic features, e.g., such as speech rate and pauses/silence. Next, we employed an LSTM based framework in order to integrate the disfluency-based and prosodic features with time sequential acoustic features. The experimental evaluation results of those integrated diverse features indicate that time sequential acoustic features contribute to improving the model with disfluency-based and prosodic features when detecting fluent speech, but not when detecting disfluent speech. Furthermore, when detecting disfluent speech, the model without time sequential acoustic features performs best even without word fragments features, but only with filled pauses and prosodic features. Keywords: automatic fluency evaluation, disfluency, filled pauses, prosody, acoustics, LSTM, spontaneous speech 1. Introduction Presentation ability has become important in various scenarios. For example, relative to a university lec- turer’s talk, speaking skill affects student understand- ing (Nishizaki et al., 2007; Kobayashi et al., 2008), and the evaluation of speech fluency of second language learners is important relative to measuring proficiency in second language learning (Mao et al., 2019). Therefore, an auto- matic speech fluency evaluation method is required. To date, several studies examined the evaluation of speech flu- ency; however, most of these studies focused on second language learning (Mao et al., 2019; Fontan et al., 2018; van Dalen et al., 2015). Among the numerous issues related to au- tomatic fluency evaluation, this paper fo- cuses on disfluency (Mahesha and Vinod, 2012; Kjellgren and Nordstrom, 2016; Deshmukh et al., 2009; Mao et al., 2019; Tohyama and Matsubara, 2006; Rasipuram et al., 2016). Although we regularly ob- serve a variety of different disfluencies in spontaneous speech, we focus on two types of phenomena, i.e., filled pauses and word fragments. With respect to filled pauses and word fragments, previous studies into automatic fluency/disfluency evaluation (Mahesha and Vinod, 2012; Kjellgren and Nordstrom, 2016; Deshmukh et al., 2009; Mao …

Content truncated to 500 words