Grammatical Templates: Improving Text Difficulty Evaluation for Language Learners

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1692–1702, Osaka, Japan, December 11-17 2016. Grammatical Templates: Improving Text Difficulty Evaluation for Language Learners Shuhan Wang Department of Computer Science Cornell University forsona@cs.cornell.edu Erik Andersen Department of Computer Science Cornell University eland@cs.cornell.edu Abstract Language students are most engaged while reading texts at an appropriate difficulty level. How- ever, existing methods of evaluating text difficulty focus mainly on vocabulary and do not pri- oritize grammatical features, hence they do not work well for language learners with limited knowledge of grammar. In this paper, we introduce grammatical templates, the expert-identified units of grammar that students learn from class, as an important feature of text difficulty evalu- ation. Experimental classification results show that grammatical template features significantly improve text difficulty prediction accuracy over baseline readability features by 7.4%. Moreover, we build a simple and human-understandable text difficulty evaluation approach with 87.7% ac- curacy, using only 5 grammatical template features. Keywords text difficulty evaluation, education, grammatical templates, language learners. 1 Introduction Evaluating text difficulty, or text readability, is an important topic in natural language processing and applied linguistics (Zamanian and Heydari, 2012; Pitler and Nenkova, 2008; Fulcher, 1997). A key challenge of text difficulty evaluation is that linguistic difficulty arises from both vocabulary and gram- mar (Richards and Schmidt, 2013). However, most existing tools either do not sufficiently take the impact of grammatical difficulty into account (Smith III et al., 2014; Sheehan et al., 2014), or use tradi- tional syntactic features, which differ from what language students actually learn, to estimate grammat- ical complexity (Schwarm and Ostendorf, 2005; Heilman et al., 2008; Franc¸ois and Fairon, 2012). In fact, language courses introduce grammar constructs together with vocabulary, and grammar constructs vary in frequency and difficulty just like vocabulary (Blyth, 1997; Manzanares and L´opez, 2008; Waara, 2004). Ideally, we would like to have better ways of estimating the grammatical complexity of a sentence. To make progress in this direction, we introduce grammatical templates as an important feature in text difficulty evaluation. These templates are what language teachers and linguists have identified as the most important units of grammatical understanding at different levels, and what students actually learn in language lessons. We also demonstrate that grammatical templates can be automatically extracted from the dependency-based parse tree of a sentence. To evaluate, we compare the difficulty prediction accuracy of grammatical templates with existing readability features in Japanese language placement tests and textbooks. Our results show that grammat- ical template features slightly outperform existing readability features. Moreover, adding grammatical template features into existing readability features significantly improves the accuracy by 7.4%. We also propose a multilevel linear classification algorithm using only 5 grammatical features. We demonstrate that this simple and human-understandable algorithm effectively predicts the difficulty level of Japanese texts with 87.7% accuracy. This work is licenced under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/ 1692 2 Related Work Text difficulty evaluation has been widely studied over the past few …

Content truncated to 500 words