Do Word Embeddings Capture Spelling Variation?

Proceedings of the 28th International Conference on Computational Linguistics, pages 870–881 Barcelona, Spain (Online), December 8-13, 2020 870 Do Word Embeddings Capture Spelling Variation? Dong Nguyen♠,♥, Jack Grieve♣, ♠The Alan Turing Institute, London, UK ♥Department of Information and Computing Sciences, Utrecht University, the Netherlands ♣Department of English Language and Linguistics, University of Birmingham, UK d.p.nguyen@uu.nl, J.Grieve@bham.ac.uk Abstract Analyses of word embeddings have primarily focused on semantic and syntactic properties. However, word embeddings have the potential to encode other properties as well. In this pa- per, we propose a new perspective on the analysis of word embeddings by focusing on spelling variation. In social media, spelling variation is abundant and often socially meaningful. Here, we analyze word embeddings trained on Twitter and Reddit data. We present three analyses using pairs of word forms covering seven types of spelling variation in English. Taken together, our results show that word embeddings encode spelling variation patterns of various types to some extent, even embeddings trained using the skipgram model which does not take spelling into ac- count. Our results also suggest a link between the intentionality of the variation and the distance of the non-conventional spellings to their conventional spellings. 1 Introduction Word embeddings play a key role in many NLP systems. Unfortunately, embeddings are opaque: It is difficult to interpret the individual vector dimensions and to understand which factors contribute to the relational similarity between words nearby in space. There has been an increasing interest in understand- ing what word embeddings are encoding, for example, to understand their impact on the performance of a wide array of NLP systems (Rogers et al., 2018). Moreover, this is especially important when em- beddings are used as research objects themselves, e.g., to study biases in society over time (Garg et al., 2018). The body of work on analyzing embeddings has focused primarily on semantic and syntactic properties (Baroni et al., 2014; Gladkova et al., 2016; Levy and Goldberg, 2014; Mikolov et al., 2013b). Instead, we propose a new perspective on word embeddings by asking how spelling variation is encoded. In social media, linguistic variation is abundant (Eisenstein, 2013; Tatman, 2015). For example, non- conventional spellings for nothing are nothin, nuthin, noooothing, nithing, and so on. Much work in NLP has focused on normalizing spelling variation (Han and Baldwin, 2011; Liu et al., 2011). Similarly, recent work by Piktus et al. (2019) aims to push embeddings of misspelled words closer to the embed- dings of their conventional forms. However, these approaches can remove valuable social (Eisenstein, 2013) and semantic (Grieve et al., 2017) signals. Crucially, many non-conventional spellings are not misspellings: by deliberately deviating from conventional spelling norms, writers create social meaning (Sebba, 2007). For example, a certain spelling may be used to evoke intimacy or to index a certain region. Capturing spelling variation patterns is therefore important for many applications, e.g., when analyzing social phenomena (Nguyen et al., 2016). To illustrate, Figure 1 shows skipgram embeddings of nothing and spelling variants projected on a 2D plane …

Content truncated to 500 words