Coherence Modeling for the Automated Assessment of Spontaneous Spoken Responses

Proceedings of NAACL-HLT 2013, pages 814–819, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Coherence Modeling for the Automated Assessment of Spontaneous Spoken Responses Xinhao Wang, Keelan Evanini, Klaus Zechner Educational Testing Service 660 Rosedale Road Princeton, NJ 08541, USA xwang002,kevanini,kzechner@ets.org Abstract This study focuses on modeling discourse co- herence in the context of automated assess- ment of spontaneous speech from non-native speakers. Discourse coherence has always been used as a key metric in human scoring rubrics for various assessments of spoken lan- guage. However, very little research has been done to assess a speaker's coherence in auto- mated speech scoring systems. To address this, we present a corpus of spoken responses that has been annotated for discourse coher- ence quality. Then, we investigate the use of several features originally developed for es- says to model coherence in spoken responses. An analysis on the annotated corpus shows that the prediction accuracy for human holistic scores of an automated speech scoring system can be improved by around 10% relative after the addition of the coherence features. Fur- ther experiments indicate that a weighted F- Measure of 73% can be achieved for the au- tomated prediction of the coherence scores. 1 Introduction In recent years, much research has been conducted into developing automated assessment systems to automatically score spontaneous speech from non- native speakers with the goals of reducing the bur- den on human raters, improving reliability, and generating feedback that can be used by language learners. Various features related to different as- pects of speaking proficiency have been exploited, such as delivery features for pronunciation, proso- dy, and fluency (Strik and Cucchiarini, 1999; Chen et al., 2009; Cheng, 2011; Higgins et al., 2011), as well as language use features for vocabulary and grammar, and content features (Chen and Zechner, 2011; Xie et al., 2012). However, discourse-level features related to topic development have rarely been investigated in the context of automated speech scoring. This is despite the fact that an im- portant criterion in the human scoring rubrics for speaking assessments is the evaluation of coher- ence, which refers to the conceptual relations be- tween different units within a response. Methods for automatically assessing discourse coherence in text documents have been widely studied in the context of applications such as natu- ral language generation, document summarization, and assessment of text readability. For example, Foltz et al. (1998) measured the overall coherence of a text by utilizing Latent Semantic Analysis (LSA) to calculate the semantic relatedness be- tween adjacent sentences. Barzilay and Lee (2004) introduced an HMM-based model for the docu- ment-level analysis of topics and topic transitions. Barzilay and Lapata (2005; 2008) presented an approach to coherence modeling which focused on the entities in the text and their grammatical transi- tions between adjacent sentences, and calculated the entity transition probabilities on the document level. Pitler et al. (2010) provided a summary of the performance of several different types of features for automated coherence evaluation, such as cohesive devices, adjacent sentence …

Content truncated to 500 words