Automatic Assessment Of Spoken English Proficiency Based on Multimodal and Multitask Transformers

Proceedings of Recent Advances in Natural Language Processing, pages 769–776 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_083 769 Automatic Assessment Of Spoken English Proficiency Based On Multimodal & Multitask Transformers Kamel Nebhi, Gyorgy Szaszak Education First Selnaustrasse 30 8001 Zürich, Switzerland {kamel.nebhi, szaszak.gyorgy}@gmail.com Abstract This paper describes technology developed to automatically grade students on their English spontaneous spoken language proficiency with common european framework of reference for languages (CEFR) level. Our automated as- sessment system contains two tasks: elicited imitation and spontaneous speech assessment. Spontaneous speech assessment is a challeng- ing task that requires evaluating various aspects of speech quality, content, and coherence. In this paper, we propose a multimodal and mul- titask transformer model that leverages both audio and text features to perform three tasks: scoring, coherence modeling, and prompt rel- evancy scoring. Our model uses a fusion of multiple features and multiple modality atten- tion to capture the interactions between audio and text modalities and learn from different sources of information. 1 Introduction Language proficiency testing is an increasingly im- portant part of our society. The need to demonstrate language abilities through standardized testing is required in many situations for access to higher education and employment opportunities. This paper presents an automatic system to ad- dress the assessment of English spoken proficiency with CEFR level. Our framework contains two tasks: elicited imitation and spontaneous speech assessment. The elicited imitation task taps into reading and speaking skills by requiring examinees to say a sentence out loud. Test takers must be able to process the input and are evaluated on their flu- ency, accuracy, and ability to use complex language orally (Van Moere, 2012). We employ statistical machine learning (ML) and natural language pro- cessing (NLP) using a transformer-based classifier to directly estimate item difficulties for a large item bank. For spontaneous speech assessment, the candi- dates are asked to talk about a prompt/question- related topic. Our spontaneous speech system is based on EF Standard English Test (EFSET) dataset. In the proposed system, the students’ spo- ken answers are first transcribed by a state-of-the- art automatic speech recognition (ASR) system and then scored using a multimodal and multitask framework. This work argues that audio and text features are complementary for a valid automatic spoken assessment system (Mayfield and Black, 2020; Gretter et al., 2019). The contributions in this paper are threefold: 1) we propose the use of test items for elicited imita- tion that can be automatically created and graded using a BERT transformer; 2) a multimodal and multitask framework for spontaneous speech as- sessment combining audio and text is proposed; 3) a complete automated assessment framework was built and evaluated using a calibrated dataset. In the pages that follow, we first summarize the state-of-the-art in automated speech assessment and then describe our approach to assess language proficiency. We then present evidence for the va- lidity and reliability of our approach using EFSET validation set and a calibration dataset. Finally, we will give a conclusion. 2 Related Works A number of approaches …

Content truncated to 500 words