Towards Effective Tutorial Feedback for Explanation Questions: A Dataset and Baselines

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 200–210, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Towards Effective Tutorial Feedback for Explanation Questions: A Dataset and Baselines Myroslava O. Dzikovska∗ and Rodney D. Nielsen† and Chris Brew‡ ∗School of Informatics, University of Edinburgh, Edinburgh, EH8 9AB, UK †Computational Language & Education Research Center University of Colorado at Boulder, Boulder, CO 80309-0594, USA ‡Educational Testing Service, Princeton, NJ 08451, USA m.dzikovska@ed.ac.uk, rodney.nielsen@colorado.edu, cbrew@ets.org Abstract We propose a new shared task on grading stu- dent answers with the goal of enabling well- targeted and flexible feedback in a tutorial di- alogue setting. We provide an annotated cor- pus designed for the purpose, a precise speci- fication for a prediction task and an associated evaluation methodology. The task is feasible but non-trivial, which is demonstrated by cre- ating and comparing three alternative baseline systems. We believe that this corpus will be of interest to the researchers working in tex- tual entailment and will stimulate new devel- opments both in natural language processing in tutorial dialogue systems and textual entail- ment, contradiction detection and other tech- niques of interest for a variety of computa- tional linguistics tasks. 1 Introduction In human-human tutoring, it is an effective strategy to ask students to explain instructional material in their own words. Self-explanation (Chi et al., 1994) and contentful talk focused on the domain are cor- related with better learning outcomes (Litman et al., 2009; Chi et al., 1994). There has therefore been much interest in developing automated tutorial dia- logue systems that ask students open-ended expla- nation questions (Graesser et al., 1999; Aleven et al., 2001; Jordan et al., 2006; VanLehn et al., 2007; Nielsen et al., 2009; Dzikovska et al., 2010a). In order to do this well, it is not enough to simply ask the initiating question, because students need the experience of engaging in meaningful dialogue about the instructional content. Thus, systems must respond appropriately to student explanations, and must provide detailed, flexible and appropriate feed- back (Aleven et al., 2002; Jordan et al., 2004). In simple domains, we can adopt a knowledge en- gineering approach and build a domain model and a diagnoser, together with a natural language parser to produce detailed semantic representations of student input (Glass, 2000; Aleven et al., 2002; Pon-Barry et al., 2004; Callaway et al., 2006; Dzikovska et al., 2010a). The advantage of this approach is that it allows for flexible adaptation of feedback to a va- riety of factors such as student performance. For example, it is easy for the system to know if the student made the same error before, and adjust its feedback to reflect it. Moreover, this approach al- lows for easy addition of new exercises : as long as an exercise relies on the concepts covered by the do- main model, the system can apply standard instruc- tional strategies to each new question automatically. However, this approach is significantly limited by the …

Content truncated to 500 words