CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3966–3977 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3966 CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations Hang Li, Wenbiao Ding, Yu Kang, Tianqiao Liu, Zhongqin Wu, Zitao Liu∗ TAL Education Group, Beijing, China {lihang4,dingwenbiao,kangyu,liutianqiao}@tal.com {wuzhongqin,liuzitao}@tal.com Abstract Existing approaches for audio-language task- specific prediction focus on building compli- cated late-fusion mechanisms. However, these models face challenges of overfitting with lim- ited labels and poor generalization. In this pa- per, we present a Cross-modal Transformer for Audio-and-Language, i.e., CTAL, which aims to learn the intra- and inter- modali- ties connections between audio and language through two proxy tasks from a large num- ber of audio-and-language pairs: masked language modeling and masked cross-modal acoustic modeling. After fine-tuning our CTAL model on multiple downstream audio- and-language tasks, we observe significant improvements on different tasks, including emotion classification, sentiment analysis, and speaker verification. Furthermore, we design a fusion mechanism in the fine-tuning phase, which allows CTAL to achieve better perfor- mance. Lastly, we conduct detailed ablation studies to demonstrate that both our novel cross-modality fusion component and audio- language pre-training methods contribute to the promising results. The code and pre- trained models are available at https:// github.com/tal-ai/CTAL_EMNLP2021. 1 Introduction Speech processing (SP) requires the understanding of a set of acoustic and language content, includ- ing phonemes, tones, words and semantic mean- ings. Different from human, who benefit from self- learning through real-world experiences, current SP methods are more like narrow experts relying heavily on a large number of task-specific human annotations and a small change may cause the learn- ing process to start all over again. In recent years, pre-training for single modality, such as language and audio signals, is widely explored due to its ease-of-use and competent generalization to vari- ous downstream tasks. ∗ The corresponding author: Zitao Liu. In the field of NLP, pre-trained models, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019) and GPT2 (Radford et al., 2019), share the same idea of first leveraging large-scale unlabeled corpus to perform contextualized language model pre-training then fine-tuning the model to adapt to downstream tasks, such as machine reading comprehension (Lai et al., 2017), question answering (Rajpurkar et al., 2016) and natural language inference (Bowman et al., 2015), etc. Following the success of pre-trained models in NLP, BERT-like models are also applied to SP community (Schneider et al., 2019; Baevski et al., 2020a,b), where robust audio representations are learned through an audio-style self-supervised context prediction task. Despite these influential unimodal methods, for tasks at the intersection of audio and language, such as speech emotion classification (Livingstone and Russo, 2018; Busso et al., 2008), speaker verifica- tion (Panayotov et al., 2015) and sentiment analy- sis (Zadeh et al., 2018), large-scale pre-training for the modality-pair of audio and language is barely explored. The previous attempt is to train task- specific experts upon the concatenation of language representations and audio representations in a late …

Content truncated to 500 words