Annotating Educational Questions for Student Response Analysis

Annotating Educational Questions for Student Response Analysis Andreea Godea and Rodney Nielsen University of North Texas 1155 Union Circle, Denton, TX, USA AndreeaGodea@my.unt.edu Rodney.Nielsen@unt.edu Abstract Questions play an important role in the educational domain, representing the main form of interaction between instructors and students. In this paper, we introduce the first taxonomy and annotated educational corpus of questions that aims to help with the analysis of student responses. The dataset can be employed in approaches that classify questions based on the expected answer types. This can be an important component in applications that require prior knowledge about the desired answer to a given question, such as educational and question answering systems. To demonstrate the applicability and the effectiveness of the data within approaches to classify questions based on expected answer types, we performed extensive experiments on our dataset using a neural network with word embeddings as features. The approach achieved a weighted F1-score of 0.511, overcoming the baseline by 12%. This demonstrates that our corpus can be effectively integrated in simple approaches that classify questions based on the response type. Keywords: question dataset, taxonomy, expected answer types, educational environment 1. Introduction Questions represent natural language sentences that express the information need of the inquirer. The analysis of ques- tions is an important part in educational systems, since questioning is the main form of interaction between instruc- tors and students. In this domain, the automatic classifica- tion of questions has multiple potential applications. For in- stance, it can help in the assessment process, in developing effective teaching strategies or in the analysis of questions asked by the students. Question datasets and taxonomies play a very important part in any automatic approach, be- ing used as a base to identify patterns in annotated data that will be applied further to unseen examples. In the educa- tional field, the existence of approaches using such data will provide important information to instructors regarding their interaction with students and will allow them to adapt their teaching strategies to the classroom’s needs. Various question taxonomies and classification approaches for educational systems have been proposed to help in the analysis of data from this domain. However, the major- ity of question datasets and taxonomies that consider the expected answer type were designed for question answer- ing (QA) and less for educational systems. This is because question classification is a very important component in QA systems, since the main goal of such systems is to identify the best possible answer among a collection of candidate answers, given a question asked by the user. On the other hand, in the educational domain, the questions have mul- tiple potential applications and can be analyzed from dif- ferent perspectives. Based on the objective being sought, researchers focused their attention on classifying questions based on their subject (Conner, 1927), the educational ob- jective (Bloom, 1956), the difficulty level (˙Ince, 2008) or the question goal (Lehnert, 1977). However, we found that this domain lacks datasets and taxonomies that aim to ana- lyze questions …

Content truncated to 500 words