Augmenting pre-trained language models with audio feature embedding for argumentation mining in political debates

Findings of the Association for Computational Linguistics: EACL 2023, pages 274–288 May 2-6, 2023 ©2023 Association for Computational Linguistics Augmenting pre-trained language models with audio feature embedding for argumentation mining in political debates Rafael Mestre†, Stuart E. Middleton, Matt Ryan, Masood Gheasi, Timothy J. Norman and Jiatong Zhu University of Southampton †r.mestre@soton.ac.uk Abstract The integration of multimodality in natural lan- guage processing (NLP) tasks seeks to exploit the complementary information contained in two or more modalities, such as text, audio and video. This paper investigates the integration of often under-researched audio features with text, using the task of argumentation mining (AM) as a case study. We take a previously reported dataset and present an audio-enhanced version (the Multimodal USElecDeb60To16 dataset). We report the performance of two text models based on BERT and GloVe embeddings, one audio model (based on CNN and Bi-LSTM) and multimodal combinations, on a dataset of 28,850 utterances. The results show that mul- timodal models do not outperform text-based models when using the full dataset. However, we show that audio features add value in fully supervised scenarios with limited data. We find that when data is scarce (e.g. with 10% of the original dataset) multimodal models yield improved performance, whereas text models based on BERT considerably decrease perfor- mance. Finally, we conduct a study with arti- ficially generated voices and an ablation study to investigate the importance of different audio features in the audio models. 1 Introduction In recent years, there has been an increasing in- terest in multimodal classification, which refers to the task of automatically classifying an input based on multiple modalities or sources of informa- tion, such as text, images and audio (Baltrušaitis et al., 2018). Multimodal approaches are benefi- cial as they can reduce the subjectivity of classi- fication with a single modality and improve the accuracy of the overall classification. However, finding the best representations (especially those that work well with other modalities), aligning and fusing them, and getting the models to co-learn are difficult challenges to overcome (Morency and Baltrušaitis, 2017). A large body of literature has focused on the combination of image and text for applications like emotion recognition (Illendula and Sheth, 2019), fake news detection (Nakamura et al., 2020), image classification (Guillaumin et al., 2010), or document image classification (Jain and Wigington, 2019). Much less attention has been paid to combining audio with text. Audio can convey a variety of information about the pitch or intonation of the speaker that can in- dicate variance in emotional state as well as better identify modes of communication like sarcasm that have been difficult for models to detect. The inte- gration of audio has successfully improved classifi- cation tasks like multimodal sentiment analysis and emotion recognition when compared with classic NLP models (Yao et al., 2020; Ho et al., 2020). In this paper, we focus on a less explored NLP area in terms of multimodality: argumentation mining (AM). AM is the computational study of arguments to develop models that can automatically identify, …

Content truncated to 500 words