Textual complexity as a predictor of difficulty of listening items in language proficiency tests

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3245–3253, Osaka, Japan, December 11-17 2016. Textual complexity as a predictor of difficulty of listening items in language proficiency tests Anastassia Loukina, Su-Youn Yoon, Jennifer Sakano, Youhua Wei, Kathleen M. Sheehan Educational Testing Service 660 Rosedale Road, Princeton, NJ 08541, USA {aloukina,syoon,jsakano,ywei,ksheehan}@ets.org Abstract In this paper we explore to what extent the difficulty of listening items in an English language proficiency test can be predicted by the textual properties of the item text. We show that a system based on multiple text complexity features can predict item difficulty for several different item types and for some items achieves higher accuracy than human estimates of item difficulty. 1 Introduction Many language proficiency tests measuring listening or reading comprehension consist of multiple ques- tions or “items” of varying complexity. For simpler items, finding the correct answer may be easy even for relatively low-proficiency speakers while other, more difficult, items require higher levels of profi- ciency. In addition, multiple sets of items or “forms” are usually created for large-scale language tests to reduce the possibility that the test takers are already familiar with a particular item. In order to be fair to all test takers, such forms need to be comparable in terms of difficulty of the items they contain. Therefore item difficulty is a crucial parameter for every new item added to the test. In test theory, item difficulty is defined by the proportion of the test takers who answer the item correctly (Holland and Thayer, 1985). This value can be estimated empirically by a pilot study before the item is used in an actual test. However, reliable estimates of item difficulty require a substantial amount of test taker re- sponses. There are also concerns about item exposure, especially if an item is to be used in a high-stakes test. In this study we consider an automated system for item difficulty prediction that can help obtain quick estimates of item difficulty for new items and assist test developers in creating items of varying complexity while retaining form comparability. The novel contribution of this study is the application of the state-of-the-art findings related to predicting the difficulty of spoken texts to predicting the difficulty of listening items in a language proficiency test. Listening comprehension items often consist of a recorded passage followed by a printed or recorded multiple choice question related to this passage. Several studies have investigated the factors that affect the difficulty of such listening and reading test items. Previous work on item difficulty prediction (Boldt and Freedle, 1996; Freedle and Kostin, 1996; Nissan et al., 1995; Rupp et al., 2001) identified three types of variables that affect item difficulty. The first category are the variables related to the text such as its length or information density. The second group of variables consider various properties of the question, for example, the number of negatives in correct and incorrect responses (“distractors”) to the question or lexical overlap between …

Content truncated to 500 words