Personalized Multimodal Feedback Generation in Education

Proceedings of the 28th International Conference on Computational Linguistics, pages 1826–1840 Barcelona, Spain (Online), December 8-13, 2020 1826 Personalized Multimodal Feedback Generation in Education Haochen Liu1, Zitao Liu2∗, Zhongqin Wu2 and Jiliang Tang1 1 Michigan State University, East Lansing, MI, USA 2 TAL Education Group, Beijing, China liuhaoc1@msu.edu, liuzitao@100tal.com, wuzhongqin@100tal.com, tangjili@msu.edu Abstract The automatic feedback of school assignments is an important application of AI in education. In this work, we focus on the task of personalized multimodal feedback generation, which aims to generate personalized feedback for teachers to evaluate students’ assignments involving multimodal inputs such as images, audios, and texts. This task involves the representation and fusion of multimodal information and natural language generation, which presents the challenges from three aspects: (1) how to encode and integrate multimodal inputs; (2) how to generate feedback specific to each modality; and (3) how to fulfill personalized feedback generation. In this paper, we propose a novel Personalized Multimodal Feedback Generation Network (PMFGN) armed with a modality gate mechanism and a personalized bias mechanism to address these challenges. Extensive experiments on real-world K-12 education data show that our model significantly outperforms baselines by generating more accurate and diverse feedback. In addition, detailed ablation experiments are conducted to deepen our understanding of the proposed framework. 1 Introduction In recent years, oral presentations have become a popular form of assignments in online K-12 education (Liu et al., 2020b). An oral presentation assignment requires students to answer a question or explain a concept verbally. It is able to test students’ oral expression skills, language organization abilities, and understanding of the topic itself simultaneously (Liu et al., 2019). Oral assignments are usually submitted in video format, which involves multimodal information. For example, given a math question “Please describe the procedure of finding the greatest common divisor of two integers”, a student is asked to record a video in which he or she presents the answer to teachers. The video contains information on the modality of image (pictures of the video), audio (voice of the student), and text (transcribed speech). To evaluate such oral presentation assignments, teachers on the online education platform provide short textual feedback. An example is shown in Figure 1. Feedback may involve different aspects of modalities, such as the clarity of video, the fluency of voice, and the relevance between the answers and the questions. Meanwhile, different teachers tend to write feedback on their own language styles. In order to lighten the workloads of teachers and improve the efficiency of online teaching, in this paper, we aim to automatically generate personalized feedback from multimodal oral presentation assignments based on the language styles of various teachers. The task, known as personalized multimodal feedback generation, is an important but rarely touched application of AI in Education (Liu et al., 2020a). One traditional solution of this task is to construct a pipeline-based system. Firstly, raw information from different modalities is passed into a series of pre-trained models for image recognition, speech fluency evaluation, and text relevance assessment. Then, …

Content truncated to 500 words