Discriminative Joint Modeling of Lexical Variation and Acoustic Confusion for Automated Narrative Retelling Assessment

Proceedings of NAACL-HLT 2013, pages 211–220, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Discriminative Joint Modeling of Lexical Variation and Acoustic Confusion for Automated Narrative Retelling Assessment Maider Lehr†, Izhak Shafran†, Emily Prud’hommeaux◦ and Brian Roark† †Center for Spoken Language Understanding, Oregon Health & Science University ◦Center for Language Sciences, University of Rochester {maiderlehr,zakshafran,emilpx,roarkbr}@gmail.com Abstract Automatically assessing the fidelity of a retelling to the original narrative – a task of growing clinical importance – is challenging, given extensive paraphrasing during retelling along with cascading automatic speech recog- nition (ASR) errors. We present a word tag- ging approach using conditional random fields (CRFs) that allows a diversity of features to be considered during inference, including some capturing acoustic confusions encoded in word confusion networks. We evaluate the approach under several scenarios, including both supervised and unsupervised training, the latter achieved by training on the output of a baseline automatic word-alignment model. We also adapt the ASR models to the domain, and evaluate the impact of error rate on per- formance. We find strong robustness to ASR errors, even using just the 1-best system out- put. A hybrid approach making use of both au- tomatic alignment and CRFs trained tagging models achieves the best performance, yield- ing strong improvements over using either ap- proach alone. 1 Introduction Narrative production tasks are an essential compo- nent of many standard neuropsychological test bat- teries. For example, narration of a wordless pic- ture book is part of the Autism Diagnostic Obser- vation Schedule (ADOS) (Lord et al., 2002) and retelling of previously narrated stories is part of both the Developmental Neuropsychological Assessment (NEPSY) (Korkman et al., 1998) and the Wech- sler Logical Memory (WLM) test (Wechsler, 1997). Such tests also arise in reading comprehension, sec- ond language learning and other computer-based tu- toring systems (Xie et al., 2012; Zhang et al., 2008). The accuracy of automated scoring of a narrative retelling depends on correctly identifying which of the source narrative’s propositions or events (what we will call ‘story elements’) have been included in the retelling. Speakers may choose to relate these elements using diverse words or phrases, and an automated method of identifying these elements needs to model the permissible variants and para- phrasings. In previous work (Lehr et al., 2012; Prud’hommeaux and Roark, 2012; Prud’hommeaux and Roark, 2011), we developed models based on automatic word-alignment methods, as described briefly in Section 3. Such alignments are learned in an unsupervised manner from a parallel corpus of manual or ASR transcripts of retellings and the orig- inal source narrative, much as in machine translation training. Relying on manual transcripts to train the align- ment models limits the ability of these methods to handle ASR errors. By instead training on ASR transcripts, these methods can automatically capture some regularities of lexical variants and their com- mon realizations by the recognizer. Additionally, ev- idence of acoustic confusability is available in word lattice output from the recognizer, which can be ex- ploited to yield more robust …

Content truncated to 500 words