MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 791–800 August 1–6, 2021. ©2021 Association for Computational Linguistics 791 MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training Mingliang Zeng, Xu Tan∗, Rui Wang, Zeqian Ju, Tao Qin, Tie-Yan Liu Microsoft Research Asia {v-minzeng,xuta,ruiwa,v-zeju,taoqin,tyliu}@microsoft.com Abstract Symbolic music understanding, which refers to the understanding of music from the sym- bolic data (e.g., MIDI format, but not audio), covers many music applications such as genre classification, emotion classification, and mu- sic pieces matching. While good music rep- resentations are beneficial for these applica- tions, the lack of training data hinders repre- sentation learning. Inspired by the success of pre-training models in natural language pro- cessing, in this paper, we develop MusicBERT, a large-scale pre-trained model for music un- derstanding. To this end, we construct a large-scale symbolic music corpus that con- tains more than 1 million music songs. Since symbolic music contains more structural (e.g., bar, position) and diverse information (e.g., tempo, instrument, and pitch), simply adopt- ing the pre-training techniques from NLP to symbolic music only brings marginal gains. Therefore, we design several mechanisms, in- cluding OctupleMIDI encoding and bar-level masking strategy, to enhance pre-training with symbolic music data. Experiments demon- strate the advantages of MusicBERT on four music understanding tasks, including melody completion, accompaniment suggestion, genre classification, and style classification. Abla- tion studies also verify the effectiveness of our designs of OctupleMIDI encoding and bar- level masking strategy in MusicBERT. 1 Introduction Music understanding, including tasks like genre classification, emotion classification, music pieces matching, has attracted lots of attention in both academia and industry. A better understanding of melody, rhythm, and music structure is not only beneficial for music information retrieval (Casey et al., 2008) but also helpful for music genera- ∗Corresponding author: Xu Tan, xuta@microsoft.com tion (Huang et al., 2018; Sheng et al., 2020). Sim- ilar to natural language, music is usually repre- sented in symbolic data format (e.g., MIDI) (Jack- endoff, 2009; McMullen and Saffran, 2004) with se- quential tokens, and some methods (Mikolov et al., 2013a,b) from NLP can be adopted for symbolic music understanding. Since the labeled training data for each music understanding task is usually scarce, previous works (Liang et al., 2020; Chuan et al., 2020) leverage unlabeled music data to learn music token embeddings, similar to word embed- dings in natural language tasks. Unfortunately, due to their shallow structures and limited unlabeled data, such embedding-based approaches have lim- ited capability to learn powerful music representa- tions. In recent years, pre-trained language models (e.g., BERT) have been verified to be powerful for representation learning from large-scale unlabeled text corpora (Devlin et al., 2018; Radford et al., 2019; Yang et al., 2019; Song et al., 2019; Brown et al., 2020; Song et al., 2020). However, it is challenging to directly apply the pre-training tech- niques from NLP to symbolic music because of the difference between natural text data and symbolic music data. First, since music songs are more struc- tural (e.g., bar, position) and diverse (e.g., tempo, …

Content truncated to 500 words