Computing and Evaluating Syntactic Complexity Features for Automated Scoring of Spontaneous Non-Native Speech

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 722–731, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Computing and Evaluating Syntactic Complexity Features for Automated Scoring of Spontaneous Non-Native Speech Miao Chen Klaus Zechner School of Information Studies NLP & Speech Group Syracuse University Educational Testing Service Syracuse, NY, USA Princeton, NJ, USA mchen14@syr.edu kzechner@ets.org Abstract This paper focuses on identifying, extracting and evaluating features related to syntactic complexity of spontaneous spoken responses as part of an effort to expand the current feature set of an automated speech scoring system in order to cover additional aspects considered important in the construct of communicative competence. Our goal is to find effective features, se- lected from a large set of features proposed previously and some new features designed in analogous ways from a syntactic complexity perspective that correlate well with human rat- ings of the same spoken responses, and to build automatic scoring models based on the most promising features by using machine learning methods. On human transcriptions with manually annotated clause and sentence boundaries, our best scoring model achieves an overall Pearson correlation with human rater scores of r=0.49 on an unseen test set, whereas correlations of models using sentence or clause boundaries from automated classifiers are around r=0.2. 1 Introduction Past efforts directed at automated scoring of speech have used mainly features related to fluen cy (e.g., speaking rate, length and distribution of pauses), pronunciation (e.g., using log-likelihood scores from the acoustic model of an Automatic Speech Recognition (ASR) system), or prosody (e.g., information related to pitch contours or syl- lable stress) (e.g., Bernstein, 1999; Bernstein et al., 2000; Bernstein et al., 2010; Cucchiarini et al., 1997; Cucchiarini et al., 2000; Franco et al., 2000a; Franco et al., 2000b; Zechner et al., 2007, Zechner et al., 2009). While this approach is a good match to most of the important properties related to low entropy speech (i.e., speech which is highly predictable), such as reading a passage aloud, it lacks many im- portant aspects of spontaneous speech which are relevant to be evaluated both by a human rater and an automated scoring system. Examples of such aspects of speech, which are considered part of the construct1 of “communicative competence (Bach- man, 1990), include grammatical accuracy, syntac- tic complexity, vocabulary diversity, and aspects of spoken discourse structure, e.g., coherence and cohesion. These different aspects of speaking pro- ficiency are often highly correlated in a non-native speaker (Xi and Mollaun, 2006; Bernstein et al., 2010), and so scoring models built solely on fea- tures of fluency and pronunciation may achieve reasonably high correlations with holistic human rater scores. However, it is important to point out that such systems would still be unable to assess many important aspects of the speaking construct and therefore cannot be seen as ideal from a validi- ty point of view.2 The purpose of this paper is to address one of these important aspects of spoken language in more detail, namely syntactic …

Content truncated to 500 words