Automatic Labeling of Problem-Solving Dialogues for Computational Microgenetic Learning Analytics

Automatic Labeling of Problem-Solving Dialogues for Computational Microgenetic Learning Analytics Yuanliang Meng∗, Anna Rumshisky∗, Florence Sullivan†, Kevin Keith† ∗Text Machine Lab, Department of Computer Science, University of Massachusetts Lowell †College of Education, University of Massachusetts Amherst {ymeng, arum}@cs.uml.edu fsullivan@educ.umass.edu, kevinkeith@landmark.edu Abstract This paper presents a recurrent neural network model to automate the analysis of students’ computational thinking in problem-solving dialogue. We have collected and annotated dialogue transcripts from middle school students solving a robotics challenge, and each dialogue turn is assigned a code. We use sentence embeddings and speaker identities as features, and experiment with linear chain CRFs and RNNs with a CRF layer (LSTM-CRF). Both the linear chain CRF model and the LSTM-CRF model outperform the na¨ıve baselines by a large margin, and LSTM-CRF has an edge between the two. To our knowledge, this is the first study on dialogue segment annotation using neural network models. This study is also a stepping-stone to automating the microgenetic analysis of cognitive interactions between students. Keywords: dialogue act annotation, recurrent neural networks, microgenetic learning analytics 1. Introduction Microgenetic analysis is an observational research tech- nique in which the researcher attends closely to the dis- course interactions and the use of tools within the learning environment in order to understand the genesis (or the ori- gins) of cognitive change (Wertsch, 1991). It has a broad impact on the design and enactment of curriculum, the de- sign of learning environments, as well as pedagogical prac- tices. Microgenetic Learning Analytics (MLA) (Sullivan et al., 2015) requires the collection and analysis of all in- teractions among students over a given period of time. It has been noted that the robustness of microgenetic analy- sis derives from high-density observation (Siegler, 2006). However, it is the time-consuming nature of collecting and analyzing high-density observations that has restricted the application of microgenetic analysis to small case studies. This paper is an initial attempt to identify computational methods that will allow for automation of MLA of conver- sational data. Such automation will transform educational researchers’ ability to perform microgenetic analysis. This paper presents a pilot study to automate the analysis of cognitive interactions in dialogue. We have transcripts of the dialogues from middle school students solving a robotics challenge (Sullivan et al., 2015). Dialogue turns (utterances) are annotated with carefully designed labels, known as the “computational thinking codes (CT codes)”. We need to build computational models to assign the codes to dialogue turns automatically. In a broader sense, this is a dialogue action (DA) annotation task, with multiple agents in the dialogue. It is also related to the sequential sentence labeling task, in which a series of sentences need to be la- beled. 2. Related Work Many different classification algorithms have been used for dialogue act annotation. The most popular ones use sup- port vector machines (SVM) (Margolis et al., 2010; Tavafi et al., 2013), Hidden Markov models (HMM) (Kim et al., 2010; Tavafi et al., 2013), conditional random fields (CRF) (Kim et al., 2010; Tavafi et al., 2013), and …

Content truncated to 500 words