Manipulating the Difficulty of C-Tests

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 360–370 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 360 Manipulating the Difficulty of C-Tests Ji-Ung Lee and Erik Schwan and Christian M. Meyer Ubiquitous Knowledge Processing (UKP) Lab and Research Training Group AIPHES Computer Science Department, Technische Universit¨at Darmstadt, Germany https://www.ukp.tu-darmstadt.de Abstract We propose two novel manipulation strategies for increasing and decreasing the difficulty of C-tests automatically. This is a crucial step towards generating learner-adaptive exercises for self-directed language learning and prepar- ing language assessment tests. To reach the desired difficulty level, we manipulate the size and the distribution of gaps based on abso- lute and relative gap difficulty predictions. We evaluate our approach in corpus-based experi- ments and in a user study with 60 participants. We find that both strategies are able to generate C-tests with the desired difficulty level. 1 Introduction Learning languages is of utmost importance in an international society and formulated as a major po- litical goal by institutions such as the European Council, who called for action to “teaching at least two foreign languages” (EC, 2002, p. 20). But also beyond Europe, there is a huge demand for lan- guage learning worldwide due to increasing global- ization, digital communication, and migration. Among multiple different learning activities re- quired for effective language learning, we study one particular type of exercise in this paper: C- tests are a special type of cloze test in which the second half of every second word in a given text is replaced by a gap (Klein-Braley and Raatz, 1982). Figure 1 (a) shows an example. To provide context, the first and last sentences of the text do not contain any gaps. C-tests rely on the reduced redundancy principle (Spolsky, 1969) arguing that a language typically employs more linguistic information than theoretically necessary to communicate unambigu- ously. Proficient speakers intuitively understand an utterance even if the level of redundancy is reduced (e.g., when replacing a word’s suffix with a gap), whereas learners typically rely on the redundant signal to extrapolate the meaning of an utterance. Besides general vocabulary knowledge, C-tests require orthographic, morphologic, syntactic, and semantic competencies (Chapelle, 1994) to cor- rectly fill in all gaps, which make them a frequently used tool for language assessment (e.g., placement tests). Given that C-tests can be easily generated automatically by introducing gaps into an arbitrary text and that there is usually only a single correct answer per gap given its context, C-tests are also relevant for self-directed language learning and massive open online courses (MOOC), where large- scale personalized exercise generation is necessary. A crucial question for such tasks is predicting and manipulating the difficulty of a C-test. For language assessment, it is important to generate C-tests with a certain target difficulty to allow for comparison across multiple assessments. For self- directed language learning and MOOCs, it is impor- tant to adapt the difficulty to the learner’s current skill level, as an exercise should be …

Content truncated to 500 words