Predicting the Difficulty of Language Proficiency Tests

Predicting the Difficulty of Language Proficiency Tests Lisa Beinborn⋄‡, Torsten Zesch§ Iryna Gurevych⋄‡ ⋄ UKP Lab, Technische Universit¨at Darmstadt ‡ UKP Lab, German Institute for Educational Research § Language Technology Lab, University of Duisburg-Essen http://www.ukp.tu-darmstadt.de Abstract Language proficiency tests are used to evaluate and compare the progress of language learners. We present an approach for automatic difficulty predic- tion of C-tests that performs on par with human ex- perts. On the basis of detailed analysis of newly collected data, we develop a model for C-test dif- ficulty introducing four dimensions: solution dif- ficulty, candidate ambiguity, inter-gap dependency, and paragraph difficulty. We show that cues from all four dimensions contribute to C-test difficulty. 1 Introduction In a labor market that is increasingly globalized, knowl- edge of at least one foreign language is more relevant than ever before. Due to increased mobility, multilingual skills are also required for private communication as friend- ships stretch across geographical and linguistic borders. In order to provide adequate language learning support, it is important to frequently evaluate learner progress on the basis of language proficiency tests that enable a fair comparison between learners. The test difficulty needs to match the intended target group as the test should be challenging for the learner but not lead to frustration. According to Vygotsky’s zone of proximal development (Vygotsky, 1978), the range of suitable material is very small. Thus, creating a test that fits this narrow target zone is a tedious and time- consuming task. Teachers predict the difficulty of a test based on their teaching experience. However, as they al- ready know the solutions, they cannot always anticipate the confusion a test might cause for learners. This results in a subjective difficulty estimation that often lacks the consistency required for comparing learners over differ- ent tests. The underlying principle of most language proficiency tests is the concept of reduced redundancy testing (Spol- sky, 1969). It is based on the idea that “natural language is redundant” and that more advanced learners can be dis- tinguished from beginners by their ability to deal with re- duced redundancy. For language testing, redundancy can be reduced by eliminating words from a text and asking the learner to fill in the gap, also known as the cloze test. The C-test is a variant of the cloze test which contains more gaps but provides part of the solution as a hint and has been found to be a good estimate for language profi- ciency (Eckes and Grotjahn, 2006). We present an approach for determining the difficulty of C-tests that overcomes the mentioned drawbacks of subjective evaluation by teachers. Our approach is based on objective measurable properties and thus produces consistent results. We show that our approach performs on par with human experts and analyze to which extent C- test difficulty is determined by individual gap properties (micro-level processing) and higher level dependencies (macro-level processing). On the theoretical level, our model provides new insights into the factors that affect difficulty in reduced redundancy testing. On the …

Content truncated to 500 words