Shallow Analysis Based Assessment of Syntactic Complexity for Automated Speech Scoring

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1305–1315, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Shallow Analysis Based Assessment of Syntactic Complexity for Automated Speech Scoring Suma Bhat Beckman Institute, University of Illinois, Urbana, IL spbhat2@illinois.edu Huichao Xue Dept. of Computer Science University of Pittsburgh Pittsburgh, PA hux10@cs.pitt.edu Su-Youn Yoon Educational Testing Service Princeton, NJ syoon@ets.org Abstract Designing measures that capture various aspects of language ability is a central task in the design of systems for auto- matic scoring of spontaneous speech. In this study, we address a key aspect of lan- guage proficiency assessment – syntactic complexity. We propose a novel measure of syntactic complexity for spontaneous speech that shows optimum empirical per- formance on real world data in multiple ways. First, it is both robust and reliable, producing automatic scores that agree well with human rating compared to the state- of-the-art. Second, the measure makes sense theoretically, both from algorithmic and native language acquisition points of view. 1 Introduction Assessment of a speaker’s proficiency in a second language is the main task in the domain of au- tomatic evaluation of spontaneous speech (Zech- ner et al., 2009). Prior studies in language ac- quisition and second language research have con- clusively shown that proficiency in a second lan- guage is characterized by several factors, some of which are, fluency in language production, pro- nunciation accuracy, choice of vocabulary, gram- matical sophistication and accuracy. The design of automated scoring systems for non-native speaker speaking proficiency is guided by these studies in the choice of pertinent objective measures of these key aspects of language proficiency. The focus of this study is the design and per- formance analysis of a measure of the syntactic complexity of non-native English responses for use in automatic scoring systems. The state-of- the art automated scoring system for spontaneous speech (Zechner et al., 2009; Higgins et al., 2011) currently uses measures of fluency and pronuncia- tion (acoustic aspects) to produce scores that are in reasonable agreement with human-rated scores of proficiency. Despite its good performance, there is a need to extend its coverage to higher order as- pects of language ability. Fluency and pronunci- ation may, by themselves, already be good indi- cators of proficiency in non-native speakers, but from a construct validity perspective1, it is neces- sary that an automatic assessment model measure higher-order aspects of language proficiency. Syn- tactic complexity is one such aspect of proficiency. By “syntactic complexity”, we mean a learner’s ability to use a wide range of sophisticated gram- matical structures. This study is different from studies that fo- cus on capturing grammatical errors in non-native speakers (Foster and Skehan, 1996; Iwashita et al., 2008). Instead of focusing on grammatical errors that are found to be highly representative of lan- guage proficiency, our interest is in capturing the range of forms that surface in language production and the degree of sophistication of such forms, collectively referred to as syntactic complexity in (Ortega, …

Content truncated to 500 words