Towards a music-language mapping

Towards a music-language mapping Michele Berlingerio, Francesca Bonin IBM Research Ireland {mberling, fbonin}[at] ie.ibm.com Abstract We explore a novel research idea, that we call Musical Language Processing (MLP), which investigates the possibility of a musical input to speech interaction systems. We present the first attempts at finding a mapping between musical pieces and dialogues, based on the frequency of musical patterns. Our findings on one possible alignment between classical piano compositions and dialogues from popular TV series are encouraging, and open the way to further investigations along this line of research. Keywords: Language generation, lexical analysis, data-driven methods 1. Introduction Natural Language Processing applications are becoming more and more pervasive in our every day lives. Dialogue systems like Amazon Echo, Google Home, Microsoft Cor- tana, or Apple Siri, are assisting us in many tasks like searches, purchases, simple calculations, etc. As of to- day, all these technologies are language-dependent, must be re-trained for each different language and they imply the knowledge of the input language for any form of com- munication. In this work, we investigate the possibility of musical input to dialogue systems. While we appreci- ate that musical input certainly requires the knowledge of the basis of musical theory for allowing communication, it presents several advantages: i) it is universal among dif- ferent languages; ii) it would facilitate the communication with dialogue system for persons with linguistics disabili- ties, that have been proven to have particular musical skills (Heaton et al., 1998; Mottron et al., 2006; Happ´e, 1999); iii) it would be simple to learn for a large class of users, i.e. musicians. We call this line of research Musical Language Processing (MLP). The fist step is to find a mapping between musical pieces and dialogues, in order to investigate the possibility of an alignment. To this aim, in this preliminary work, we study one possible mapping between classical piano composi- tions and dialogues from popular TV series, by investigat- ing the frequency distributions of words in dialogues and chords in music. From this analysis we aim at creating a lexical mapping between chords and the English vocabu- lary, which would allow, at a later step, to investigate pos- sible syntactical mappings (n-chords and n-grams), and se- mantic mappings, where the musical language will be able to express simple meaningful sentences. Note that, instead of trying to map a chord to a word, and then forcing users to learn the mapping and play the right sequence of chords to form a sentence, our final goal is to “reverse-engineer” already existing music, and find a mapping that any person with “some” knowledge of mu- sic would be able to reproduce, without having to learn a new language, or a new kind of music. This is a what-if analysis of what would happen if, for example, we would map existing Bach’s compositions, chord by chord: would they form a meaningful sentence? While there are works involving music and Natural Lan- guage Processing, to the best of our knowledge this is …

Content truncated to 500 words