Annotation and Multi-modal Methods for Quality Assessment of Multi-party Discussion

Annotation and Multi-modal Methods for Quality Assessment of Multi-party Discussion Tsukasa Shiota Kyushu Institute of Technology 680-4 Kawazu Iizuka Fukuoka JAPAN Kazutaka Shimada Kyushu Institute of Technology 680-4 Kawazu Iizuka Fukuoka JAPAN shimada@ai.kyutech.ac.jp Abstract Discussion quality assessment tasks have re- cently attracted significant attention in nat- ural language processing. However, there have been few studies on challenging such tasks, with a focus on synchronous discus- sions. In this study, we annotate quality scores to each discussion in an existing multi-modal multi-party discussion corpus. Furthermore, we propose some quality assessment meth- ods with multi-modal inputs. As the results show, attention-based long short-term memory (LSTM) with multi-modal inputs produces the best performance for the “Effectiveness” crite- rion whereas text information has an important role in the “Reasonableness.” 1 Introduction In recent years, problem-based and cooperative learning have been attracting attention as a means of skills training, such as communication skills, in education. One educational training approach is a group discussion, which involves debate and consensus-building. Introducing this learning ap- proach to a classroom requires a great deal of ef- fort to evaluate and provide feedback on the abil- ities and achievements of all groups and individu- als from various perspectives because several dis- cussion groups usually exist in a single class at the same time. Furthermore, an assessment is a difficult task because there are no correct an- swers regarding discussions in general. Moreover, quantitative and objective evaluations are difficult. Therefore, an automatic assessment, such as a vi- sualization of the discussion state and a judgment of the discussion score, is a desirable and valuable task for education, examinations through discus- sion, and so forth. It will be possible to reduce the burden of evaluation activities on the evaluators. One of the educational applications in natural language processing is automated essay scoring (AES) (Ke and Ng, 2019) as an argument qual- ity evaluation. However, the structure of a spoken discussion, which is our target in this paper, is not as clear as that of a written discussion. In addi- tion, in spoken discussions, both verbal and non- verbal information have important roles in under- standing and evaluating the discussion. Mukawa et al. (2018) have reported that non-verbal fea- tures, such as gestures and an interval of utter- ances, have a powerful effect on group discus- sions. In this study, we annotate several quality assess- ment criteria and scores to a multi-modal multi- party discussion corpus. The language used is Japanese, and the corpus is freely available1. In addition, we propose the use of machine-learning- based methods, such as a support vector machine (SVM) and neural networks, and then evaluate the methods using multi-modal inputs. In the experi- ment, we discuss the relationships between the as- sessment criteria and input modalities. 2 Related work There are some dialogue and meeting corpora (Carletta, 2007; Janin et al., 2003). Some face- to-face discussion corpora have been also devel- oped. Zhang et al. (2016) have constructed a cor- pus through a competitive debate format. They reported …

Content truncated to 500 words