Pre-Training BERT on Domain Resources for Short Answer Grading

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6071–6075, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6071 Pre-Training BERT on Domain Resources for Short Answer Grading Chul Sung∗ IBM Watson Education, USA daniel.c.sung@gmail.com Tejas Indulal Dhamecha IBM Research, India tidhamecha@in.ibm.com Swarnadeep Saha† UNC Chapel Hill, USA swarna@cs.unc.edu Tengfei Ma IBM Research AI, USA tengfei.ma1@ibm.com Vinay Reddy IBM Watson Education, USA vinay.kasireddy@ibm.com Rishi Arora IBM GBS, India arishi@in.ibm.com Abstract Pre-trained BERT contextualized representa- tions have achieved state-of-the-art results on multiple downstream NLP tasks by fine-tuning with task-specific data. While there has been a lot of focus on task-specific fine-tuning, there has been limited work on improving the pre- trained representations. In this paper, we ex- plore ways of improving the pre-trained con- textual representations for the task of auto- matic short answer grading, a critical compo- nent of intelligent tutoring systems. We show that the pre-trained BERT model can be im- proved by augmenting data from the domain- specific resources like textbooks. We also present a new approach to use labeled short answering grading data for further enhance- ment of the language model. Empirical eval- uation on multi-domain datasets shows that task-specific fine-tuning on the enhanced pre- trained language model achieves superior per- formance for short answer grading. 1 Introduction Intelligent tutoring system (ITS) is one of the tools to facilitate personalized learning. Auto- matic short answer grading (ASAG) is an impor- tant component of a dialog-based tutoring (DBT) system; especially, to enable Socratic tutoring. Automatic grading is the task of evaluating the correctness of a student answer for a specific ques- tion by comparing it to a reference answer. In short answer grading, the reference answers are typi- cally one or two sentences long and close ended. A broad range of approaches from simple bag- of-words to transfer learning and deep neural net- works have been explored to address the short an- swer grading problem (Mueller and Thyagarajan, 2016; Saha et al., 2018; Marvaniya et al., 2018). ∗Corresponding author. †The work was done when the author was an employee at IBM Research, India. Pre-training BERT Updating the BERT Domain-Textbooks or Question-Answers Fine-tuning the BERT Textbooks or QAs Labeled Task Data Figure 1: In this work, we propose to update the pre- trained BERT language model by utilizing the textbook or question-answer data to improve short answer grad- ing. For a variety of natural language processing (NLP) tasks, state-of-the-art results have been re- ported with pre-trained deep language models, such as BERT (Devlin et al., 2018), GPT (Radford et al., 2018), and ULMFit (Howard and Ruder, 2018). In these approaches, pre-trained language models are utilized for down-stream NLP tasks by means of task-specific fine-tuning. A typical DBT, and therefore a short answer grading system, is often used across various sub- jects or domains (e.g., Science, Sociology, and Psychology). A pre-trained language model, such as that of BERT, is typically trained on generic …

Content truncated to 500 words