Urban Dictionary Embeddings for Slang NLP Applications

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4764–4773 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4764 Urban Dictionary Embeddings for Slang NLP Applications Steven R. Wilson1, Walid Magdy1,2, Barbara McGillivray2,5, Kiran Garimella3, Gareth Tyson2,4 1The University of Edinburgh, 2The Alan Turing Institute 3Massachusetts Institute of Technology, 4Queen Mary University of London, 5The University of Cambridge steven.wilson@ed.ac.uk, wmagdy@inf.ed.ac.uk, bmcgillivray@turing.ac.uk, garimell@mit.edu, g.tyson@qmul.ac.uk Abstract The choice of the corpus on which word embeddings are trained can have a sizable effect on the learned representations, the types of analyses that can be performed with them, and their utility as features for machine learning models. To contribute to the existing sets of pre-trained word embeddings, we introduce and release the first set of word embeddings trained on the content of Urban Dictionary, a crowd-sourced dictionary for slang words and phrases. We show that although these embeddings are trained on fewer total tokens (by at least an order of magnitude compared to most popular pre-trained embeddings), they have high performance across a range of common word embedding evaluations, ranging from semantic similarity to word clustering tasks. Further, for some extrinsic tasks such as sentiment analysis and sarcasm detection where we expect to require some knowledge of colloquial language on social media data, initializing classifiers with the Urban Dictionary Embeddings resulted in improved performance compared to initializing with a range of other well-known, pre-trained embeddings that are order of magnitude larger in size. Keywords: word embeddings, urban dictionary, slang, sentiment, sarcasm 1. Introduction Word embeddings provide useful representations of the meanings of words in the form of vectors, and have become the de-facto mapping from tokens to fixed-length continu- ous inputs for machine learning models in the field of natural language processing. These vectors can be randomly initial- ized and subsequently learned from the training data for a specific task. However, the ability to pre-train embeddings on a large corpus in an unsupervised way is a powerful technique for transferring generally applicable, word-level semantic information from these massive corpora to a wide range of downstream tasks. Especially in settings with a smaller amount of domain-specific training data, initializ- ing models with these pre-trained embeddings can provide useful representations for words that may not have been seen in the training data and help models converge more quickly during training time. As training time is reduced, the fact that these models can be trained once and subsequently shared with other researchers reduces the environmental impact of having each team of researchers recompute new embeddings, placing the paradigm of training and sharing within the principles of Green AI (Schwartz et al., 2019). When training word embeddings, there are a number of al- gorithms and parameters to choose from and tune, but it has also been shown that the choice of corpus on which to pre-train is also extremely important (Nooralahzadeh et al., 2018; Risch and Krestel, 2019). This not only affects the vo- cabulary – and therefore the …

Content truncated to 500 words