What do tokens know about their characters and how do they know it?

21229

Authors

nan

Abstract

AbstractPre-trained language models (PLMs) that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information, despite lacking explicit access to the character composition of tokens. Here, studying a range of models (e.g., GPT- J, BERT, RoBERTa, GloVe), we probe what word pieces encode about character-level information by training classifiers to predict the presence or absence of a particular alphabetical character in a token, based on its embedding (e.g., probing whether the model embedding for “cat” encodes that it contains the character “a”). We find that these models robustly encode character-level information and, in general, larger models perform better at the task. We show that these results generalize to characters from non-Latin alphabets (Arabic, Devanagari, and Cyrillic). Then, through a series of experiments and analyses, we investigate the mechanisms through which PLMs acquire English-language character information during training and argue that this knowledge is acquired through multiple phenomena, including a systematic relationship between particular characters and particular parts of speech, as well as natural variability in the tokenization of related strings.

Keywords

nan

Full Content

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 231 - 243 July 10-15, 2022 ©2022 Association for Computational Linguistics Political Ideology and Polarization: A Multi-dimensional Approach Barea Sinno1∗ Bernardo Oviedo2∗ Katherine Atwell3∗ Malihe Alikhani3 Junyi Jessy Li4 1 Political Science, Rutgers University 2 Computer Science, 4 Linguistics, The University of Texas at Austin 3 Computer Science, University of Pittsburgh barea.sinno@gmail.com, bernyoviedo@utexas.edu, kaa139@pitt.edu malihe@pitt.edu, jessy@utexas.edu Abstract Analyzing ideology and polarization is of criti- cal importance in advancing our grasp of mod- ern politics. Recent research has made great strides towards understanding the ideological bias (i.e., stance) of news media along the left- right spectrum. In this work, we instead take a novel and more nuanced approach for the study of ideology based on its left or right positions on the issue being discussed. Aligned with the theoretical accounts in political science, we treat ideology as a multi-dimensional construct, and introduce the first diachronic dataset of news articles whose ideological positions are annotated by trained political scientists and lin- guists at the paragraph level. We showcase that, by controlling for the author’s stance, our method allows for the quantitative and tempo- ral measurement and analysis of polarization as a multidimensional ideological distance. We further present baseline models for ideology prediction, outlining a challenging task distinct from stance detection. 1 Introduction Political ideology rests on a set of beliefs about the proper order of a society and ways to achieve this order (Jost et al., 2009; Adorno et al., 2019; Camp- bell et al., 1980). In Western politics, these world- views translate into a multi-dimensional construct that includes: equal opportunity as opposed to eco- nomic individualism; general respect for tradition, hierarchy and stability as opposed to advocating for social change; and a belief in the un/fairness and in/efficiency of markets (Jost et al., 2009). The divergence in ideology, i.e., polarization, is the undercurrent of propaganda and misinfor- mation (Vicario et al., 2019; Bessi et al., 2016; Stanley, 2015). It can congest essential democratic functions with an increase in the divergence of political ideologies. Defined as a growing ideo- * Equal contribution ordered by first name. Two dimensions: trade and economic liberalism The U.S. aim is to create a monetary sys- tem with enough flexibility to prevent bar- gain-hungry money from rolling around the world like loose ballast on a ship dis- rupting normal trade and currency flows. Nixon goals: dollar, trade stability. This must be accompanied, Washington says, by reduction of [trade] barriers ... One dimension: trade protectionism The controls program, which Mr. Nixon inaugurated Aug. 15, 1971, has helped to reduce inflation to about 3 percent yearly, and to boost annual U.S. eco- nomic growth to more than 7 percent... Table 1: Excerpts from news article #730567 in COHA (Davies, 2012). The first paragraph advocates for liberalism and the reduction of trade barriers. It also has a domestic economic dimension. The second para- graph, on the contrary, advocates for protectionism …

Content truncated to 500 words

Labeled Text

AbstractPre-trained language models (PLMs) that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information, despite lacking explicit access to the character composition of tokens. Here, studying a range of models (e.g., GPT- J, BERT, RoBERTa, GloVe), we probe what word pieces encode about character-level information by training classifiers to predict the presence or absence of a particular alphabetical character in a token, based on its embedding (e.g., probing whether the model embedding for “cat” encodes that it contains the character “a”). We find that these models robustly encode character-level information and, in general, larger models perform better at the task. We show that these results generalize to characters from non-Latin alphabets (Arabic, Devanagari, and Cyrillic). Then, through a series of experiments and analyses, we investigate the mechanisms through which PLMs acquire English-language character information during training and argue that this knowledge is acquired through multiple phenomena, including a systematic relationship between particular characters and particular parts of speech, as well as natural variability in the tokenization of related strings.

Publication Details

Paper ID:
21229
Year:
2022
Publication:
TIPSTER
Publication Type:
conf
DOI:
nan