Lexical variation in English language podcasts, editorial media, and social media

Lexical variation in English language podcasts, editorial media, and social media Jussi Karlgren, Spotify, Stockholm, Sweden Abstract The study presented in this paper demonstrates how transcribed podcast material difers with respect to lexical content from other collections of English language data: editorial text, social media, both long form and microblogs, dialogue from movie scripts, and transcribed phone conversations. Most of the recorded diferences are as might be expected, reflecting known or assumed diference between spoken and writen language, between dialogue and soliloquy, and between scripted formal and unscripted informal language use. Most notably, podcast material, compared to the hitherto typical training sets from editorial media, is characterised by being in the present tense, and with a much higher incidence of pronouns, interjections, and negations. These characteristics are, unsurprisingly, largely shared with social media texts. Where podcast material difers from social media material is in its atitudinal content, with many more amplifiers and much less negative atitude than in blog texts. This variation, besides being of philological interest, has ramifications for computational work. Information access for material which is not primarily topical should be designed to be sensitive to such variation that defines the data set itself and discriminates items within it. In general, training sets for language models are a non-trivial parameter which are likely to show efects both expected and unexpected when applied to data from other sources and the characteristics and provenance of data used to train a model should be listed on the label as a minimal form of downstream consumer protection. 1 Genres and podcast transcripts The way human language is used varies across chan- nels and styles, and we have for the longest while made a clear distinction between spoken and writen lan- guage as two major distinctive modes of communica- tion (Cederschi¨old, 1897; Ong, 1982; Biber, 1991; Coul- mas, 2003). The diferences between writing and reading can to a large extent be related to situational diferences: where speech has been used in transient situations in which interlocutors are present, writing has typically been used in asynchronous communication with par- ticipants at a remove from each other. This distinction has through the introduction of communication tech- nologies become less and less clear-cut. Writen lan- guage is used for momentary and fleeting conversa- tions with litle planning or editorial oversight; spoken language material is created, published and distributed in ways which are more formal and more permanent and archival than before. Podcasts are a new medium and a new format for spoken language. The styles of language use in podcasts are as yet unformed and have not yet co- alesced into stable functional and generally accepted genres: podcast material will require us to recalibrate many of the assumptions we make about how language is used. Recently, a collection of over 100,000 pod- cast episodes, including automatically generated tran- scripts, has been released for the purposes of retrieval and summarisation experimentation. The companion paper released with the podcast material set gives some indicative diferences between the transcripts …

Content truncated to 500 words