Word Association Profiles and their Use for Automated Scoring of Essays

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1148–1158, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Word Association Profiles and their Use for Automated Scoring of Essays Beata Beigman Klebanov and Michael Flor Educational Testing Service 660 Rosedale Road Princeton, NJ 08541 {bbeigmanklebanov,mflor}@ets.org Abstract We describe a new representation of the content vocabulary of a text we call word association profile that captures the pro- portions of highly associated, mildly asso- ciated, unassociated, and dis-associated pairs of words that co-exist in the given text. We illustrate the shape of the dis- tirbution and observe variation with genre and target audience. We present a study of the relationship between quality of writ- ing and word association profiles. For a set of essays written by college graduates on a number of general topics, we show that the higher scoring essays tend to have higher percentages of both highly asso- ciated and dis-associated pairs, and lower percentages of mildly associated pairs of words. Finally, we use word association profiles to improve a system for automated scoring of essays. 1 Introduction The vast majority of contemporary research that investigates statistical properties of language deals with characterizing words by extracting infor- mation about their behavior from large corpora. Thus, co-occurrence of words in n-word windows, syntactic structures, sentences, paragraphs, and even whole documents is captured in vector-space models built from text corpora (Turney and Pan- tel, 2010; Basili and Pennacchiotti, 2010; Erk and Pad´o, 2008; Mitchell and Lapata, 2008; Bullinaria and Levy, 2007; Jones and Mewhort, 2007; Pado and Lapata, 2007; Lin, 1998; Landauer and Du- mais, 1997; Lund and Burgess, 1996; Salton et al., 1975). However, little is known about typical pro- files of texts in terms of co-occurrence behavior of their words. Some information can be inferred from the success of statistical techniques in pre- dicting certain structures in text. For example, the fact that a text segmentation algorithm that uses information about patterns of word co-occurrences can detect sub-topic shifts in a text (Riedl and Bie- mann, 2012; Misra et al., 2009; Eisenstein and Barzilay, 2008) tells us that texts contain some proportion of more highly associated word pairs (those in subsequent sentences within the same topical unit) and of less highly associated pairs (those in sentences from different topical units).1 Yet, does each text have a different distribution of highly associated, mildly associated, unassoci- ated, and dis-associated pairs of words, or do texts tend to strike a similar balance of these? What are the proportions of the different levels of asso- ciation, how much variation there exists, and are there systematic differences between various kinds of texts? We present research that makes a first step in addressing these questions. From the applied perspective, our interest is in quantifying differences between well-written and poorly written essays, for the purposes of auto- mated scoring of essays. We therefore concentrate on essay data for the main experiments reported in this paper, although some additional corpora will …

Content truncated to 500 words