On the Effectiveness of Using Syntactic and Shallow Semantic Tree Kernels for Automatic Assessment of Essays

International Joint Conference on Natural Language Processing, pages 767–773, Nagoya, Japan, 14-18 October 2013. On the Effectiveness of Using Syntactic and Shallow Semantic Tree Kernels for Automatic Assessment of Essays Yllias Chali University of Lethbridge Lethbridge, AB, Canada chali@cs.uleth.ca Sadid A. Hasan University of Lethbridge Lethbridge, AB, Canada hasan@cs.uleth.ca Abstract This paper is concerned with the prob- lem of automatic essay grading, where the task is to grade student written essays given course materials and a set of human- graded essays as training data. Latent Se- mantic Analysis (LSA) has been used ex- tensively over the years to accomplish this task. However, the major limitation of LSA is that it only retains the frequency of words by disregarding the word sequence, and the syntactic and semantic structure of texts. As a remedy, we propose the use of syntactic and shallow semantic tree ker- nels for grading essays. Experiments sug- gest that syntactic and semantic structural information can significantly improve the performance of the state-of-the-art LSA- based models for automatic essay grading. 1 Introduction and Related Work To evaluate the content of free texts is a chal- lenging task for humans. Automation of this pro- cess is useful when an expert evaluator is un- available in today’s Internet-based learning envi- ronment. Research to automate the assessment of free texts, such as grading student-written es- says, has been carried out over the years (Kakko- nen et al., 2006; Kakkonen and Sutinen, 2004; Kanejiya et al., 2003; Persing et al., 2010; Yan- nakoudakis et al., 2011). Some notable essay scor- ing systems currently available are AutoScore by American Institutes for Research (AIR), Bookette by CTB/McGraw-Hill, Project Essay Grade by Measurement, Inc. and Intelligent Essay Asses- sor by Pearson Knowledge Technologies. The approaches such as Project Essay Grade and e- rater were solely based on some simple surface features that took essay-length, number of com- mas etc. into consideration (Page and Petersen, 1995; Powers et al., 2000). The major drawback of these systems is that they ignore the creativity fac- tor by only dealing with the simple measures. To overcome this limitation, recent researches tend to focus on understanding the inner meaning of the texts. Latent Semantic Analysis (LSA) (Landauer et al., 1998; Deerwester et al., 1990) has been shown to fit well in addressing this task (Kakkonen et al., 2006; Kakkonen and Sutinen, 2004; Lintean et al., 2010; Kanejiya et al., 2003). LSA uses a sophisticated approach to decode the inherent relationships between a context (typ- ically a sentence, a paragraph or a document) and the words that they contain. This approach is based on Bag-Of-Words (BOW) assumption that uses the frequency of occurrence of each word in the context to construct a word-by-context co- occurrence matrix (Kanejiya et al., 2003). The major limitation of LSA is that it only retains the frequency of the words and does not take into ac- count the sequence of them (word ordering). It ignores the syntactic and semantic structure of the context and thus, cannot distinguish …

Content truncated to 500 words