A Corpus for Modeling Word Importance in Spoken Dialogue Transcripts

A Corpus for Modeling Word Importance in Spoken Dialogue Transcripts Sushant Kafle, Matt Huenerfauth Rochester Institute of Technology 1 Lomb Memorial Drive, Rochester, NY sushant@mail.rit.edu, matt.huenerfauth@rit.edu Abstract Motivated by a project to create a system for people who are deaf or hard-of-hearing that would use automatic speech recognition (ASR) to produce real-time text captions of spoken English during in-person meetings with hearing individuals, we have augmented a transcript of the Switchboard conversational dialogue corpus with an overlay of word-importance annotations, with a numeric score for each word, to indicate its importance to the meaning of each dialogue turn. Further, we demonstrate the utility of this corpus by training an automatic word importance labeling model; our best performing model has an F-score of 0.60 in an ordinal 6-class word-importance classification task with an agreement (concordance correlation coefficient) of 0.839 with the human annotators (agreement score between annotators is 0.89). Finally, we discuss our intended future applications of this resource, particularly for the task of evaluating ASR performance, i.e. creating metrics that predict ASR-output caption text usability for DHH users better than Word Error Rate (WER). Keywords: Word Importance Annotation, Speech Recognition Evaluation, Word Importance Prediction 1. Introduction There has been increasing interest among researchers of speech and language technology applications to identify the importance of individual words, for the overall meaning of the text. Depending on the context of how the impor- tance of a word is defined, this task has found use in vari- eties of applications such as text summarization (Hong and Nenkova, 2014; Yih et al., 2007), text classification (Sheikh et al., 2016), or speech synthesis (Mishra et al., 2007). Our laboratory is currently designing a system to benefit people who are deaf or hard-of-hearing (DHH) who are en- gaged in a live meeting with hearing colleagues. In many settings, sign language interpreting or professional caption- ing (where a human types the speech, displayed as text on a screen for the user), are unavailable, e.g. in impromptu con- versations in the workplace. A system that uses automatic speech recognition (ASR) to generate captions in real-time could display this text on mobile devices for DHH users, but text output from ASR systems inevitably contains er- rors. Thus, we were motivated to understand which words in the text were most important to the overall meaning, to inform our evaluation of ASR accuracy for this task. In this paper, we present a word-importance annotation of transcripts of the Switchboard corpus (Godfrey et al., 1992). While our overall goal is to produce measures of ASR accuracy for our caption application; to demonstrate the use of this corpus, in this paper, we present models that predict word-importance in spoken dialogue transcripts. 1.1. ASR Evaluation ASR researchers generally report the performance of their systems using a metric called Word Error Rate (WER). The metric considers the number of errors in the output of the ASR system, normalized by the number of words human actually said in the audio recording. While WER has been the …

Content truncated to 500 words