Automated Japanese Essay Scoring System based on Articles Written by Experts

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 233–240, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Automated Japanese Essay Scoring System based on Articles Written by Experts Tsunenori Ishioka Research Division The National Center for University Entrance Examinations Tokyo 153-8501, Japan tunenori@rd.dnc.ac.jp Masayuki Kameda Software Research Center Ricoh Co., Ltd. Tokyo 112-0002, Japan masayuki.kameda@nts.ricoh.co.jp Abstract We have developed an automated Japanese essay scoring system called Jess. The sys- tem needs expert writings rather than ex- pert raters to build the evaluation model. By detecting statistical outliers of prede- termined aimed essay features compared with many professional writings for each prompt, our system can evaluate essays. The following three features are exam- ined: (1) rhetoric — syntactic variety, or the use of various structures in the arrange- ment of phases, clauses, and sentences, (2) organization — characteristics associ- ated with the orderly presentation of ideas, such as rhetorical features and linguistic cues, and (3) content — vocabulary re- lated to the topic, such as relevant infor- mation and precise or specialized vocabu- lary. The final evaluation score is calcu- lated by deducting from a perfect score as- signed by a learning process using editori- als and columns from the Mainichi Daily News newspaper. A diagnosis for the es- say is also given. 1 Introduction When giving an essay test, the examiner expects a written essay to reflect the writing ability of the ex- aminee. A variety of factors, however, can affect scores in a complicated manner. Cooper (1984) states that “various factors including the writer, topic, mode, time limit, examination situation, and rater can introduce error into the scoring of essays used to measure writing ability.” Most of these factors are present in giving tests, and the human rater, in particular, is a major error factor in the scoring of essays. In fact, many other factors influence the scoring of essay tests, as listed below, and much research has been devoted. � Handwriting skill (handwriting quality, spelling) (Chase, 1979; Marshall and Powers, 1969) � Serial effects of rating (the order in which es- say answers are rated) (Hughes et al., 1983) � Topic selection (how should essays written on different topics be rated?) (Meyer, 1939) � Other error factors (writer’s gender, ethnic group, etc.) (Chase, 1986) In recent years, with the aim of removing these error factors and establishing fairness, consider- able research has been performed on computer- based automated essay scoring (AES) systems (Burstein et al., 1998; Foltz et al., 1999; Page et al., 1997; Powers et al., 2000; Rudner and Liang, 2002). The AES systems provide the users with prompt feedback to improve their writings. Therefore, many practical AES systems have been used. E- rater (Burstein et al., 1998), developed by the Ed- ucational Testing Service, began being used for operational scoring of the Analytical Writing As- sessment in the Graduate Management Admis- sion Test (GMAT), an entrance examination for business graduate schools, in February 1999, and it has …

Content truncated to 500 words