Dictionary-Assisted Supervised Contrastive Learning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10217 - 10235 December 7-11, 2022 ©2022 Association for Computational Linguistics Dictionary-Assisted Supervised Contrastive Learning Patrick Y. Wu1, Richard Bonneau1,2,4,5, Joshua A. Tucker1,2,3, and Jonathan Nagler1,2,3 1 Center for Social Media and Politics, New York University 2 Center for Data Science, New York University 3 Department of Politics, New York University 4 Department of Biology, New York University 5 Courant Institute of Mathematical Sciences, New York University {pyw230, bonneau, joshua.tucker, jonathan.nagler}@nyu.edu Abstract Text analysis in the social sciences often in- volves using specialized dictionaries to rea- son with abstract concepts, such as perceptions about the economy or abuse on social media. These dictionaries allow researchers to impart domain knowledge and note subtle usages of words relating to a concept(s) of interest. We in- troduce the dictionary-assisted supervised con- trastive learning (DASCL) objective, allowing researchers to leverage specialized dictionar- ies when fine-tuning pretrained language mod- els. The text is first keyword simplified: a common, fixed token replaces any word in the corpus that appears in the dictionary(ies) relevant to the concept of interest. During fine-tuning, a supervised contrastive objective draws closer the embeddings of the original and keyword-simplified texts of the same class while pushing further apart the embeddings of different classes. The keyword-simplified texts of the same class are more textually similar than their original text counterparts, which addition- ally draws the embeddings of the same class closer together. Combining DASCL and cross- entropy improves classification performance metrics in few-shot learning settings and social science applications compared to using cross- entropy alone and alternative contrastive and data augmentation methods.1 1 Introduction We propose a supervised contrastive learning ap- proach that allows researchers to incorporate dic- tionaries of words related to a concept of interest when fine-tuning pretrained language models. It is conceptually simple, requires low computational resources, and is usable with most pretrained lan- guage models. Dictionaries contain words that hint at the sen- timent, stance, or perception of a document (see, e.g., Fei et al., 2012). Social science experts often 1Our code is available at https://github.com/ SMAPPNYU/DASCL. craft these dictionaries, making them useful when the underlying concept of interest is abstract (see, e.g., Brady et al., 2017; Young and Soroka, 2012). Dictionaries are also useful when specific words that are pivotal to determining the classification of a document may not exist in the training data. This is a particularly salient issue with small corpora, which is often the case in the social sciences. However, recent supervised machine learning ap- proaches do not use these dictionaries. We propose a contrastive learning approach, dictionary-assisted supervised contrastive learning (DASCL), that al- lows researchers to leverage these expert-crafted dictionaries when fine-tuning pretrained language models. We replace all the words in the corpus that belong to a specific lexicon with a fixed, common token. When using an appropriate dictionary, key- word simplification increases the textual similarity of documents in the same class. We then use a supervised contrastive objective to draw …

Content truncated to 500 words