Characterizing English Variation across Social Media Communities with BERT

Characterizing English Variation across Social Media Communities with BERT Li Lucy and David Bamman University of California, Berkeley {lucy3 li, dbamman}@berkeley.edu Abstract Much previous work characterizing language variation across Internet social groups has fo- cused on the types of words used by these groups. We extend this type of study by em- ploying BERT to characterize variation in the senses of words as well, analyzing two months of English comments in 474 Reddit communi- ties. The specificity of different sense clusters to a community, combined with the specific- ity of a community’s unique word types, is used to identify cases where a social group’s language deviates from the norm. We validate our metrics using user-created glossaries and draw on sociolinguistic theories to connect lan- guage variation with trends in community be- havior. We find that communities with highly distinctive language are medium-sized, and their loyal and highly engaged users interact in dense networks. 1 Introduction Internet language is often popularly characterized as a messy variant of ‘‘standard’’ language (Desta, 2014; Magalh˜aes, 2019). However, work in socio- linguistics has demonstrated that online language is not homogeneous (Herring and Paolillo, 2006; Nguyen et al., 2016; Eisenstein, 2013). Instead, it expresses immense amounts of variation, often driven by social variables. Online language con- tains lexical innovations, such as orthographic variants, but also repurposes words with new meanings (Pei et al., 2019; Stewart et al., 2017). There has been much attention on which words are used across these social groups, including work examining the frequency of types (Zhang et al., 2017; Danescu-Niculescu-Mizil et al., 2013). However, there is also increasing interest in how words are used in these online communities as well, including variation in meaning (Yang and Eisenstein, 2017; Del Tredici and Fern´andez, 2017). For example, a word such as python in Figure 1 has different usages depending on the community in which it is used. Our work exam- ines both lexical and semantic variation, and operationalizes the study of the latter using BERT (Devlin et al., 2019). Social media language is an especially inter- esting domain for studying lexical semantics because users’ word use is far more dynamic and varied than is typically captured in standard sense inventories like WordNet. Online commu- nities that sustain linguistic norms have been characterized as virtual communities of practice (Eckert and McConnell-Ginet, 1992; Del Tredici and Fern´andez, 2017; Nguyen and Ros´e, 2011). Users may develop wiki pages, or guides, for their communities that outline specific jargon and rules. However, some communities exhibit more language variation than others. One central goal in sociolinguistics is to investigate what social factors lead to variation, and how they relate to the growth and maintenance of sociolects, registers, and styles. To enable our ability to answer these types of questions from a computational perspec- tive, we must first develop metrics for measuring variation. Our work quantifies how much the language of an online community deviates from the norm and identifies communities that contain unique language varieties. We define community-specific language in …

Content truncated to 500 words