Interpreting Character Embeddings With Perceptual Representations: The Case of Shape, Sound, and Color

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6819 - 6836 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Interpreting Character Embeddings With Perceptual Representations: The Case of Shape, Sound, and Color Sidsel Boldsen and Manex Agirrezabal and Nora Hollenstein Centre for Language Technology Department of Nordic Studies and Linguistics University of Copenhagen {sbol,manex.aguirrezabal,nora.hollenstein}@hum.ku.dk Abstract Character-level information is included in many NLP models, but evaluating the infor- mation encoded in character representations is an open issue. We leverage perceptual repre- sentations in the form of shape, sound, and color embeddings and perform a representa- tional similarity analysis to evaluate their cor- relation with textual representations in five languages. This cross-lingual analysis shows that textual character representations correlate strongly with sound representations for lan- guages using an alphabetic script, while shape correlates with featural scripts. We further de- velop a set of probing classifiers to intrinsi- cally evaluate what phonological information is encoded in character embeddings. Our re- sults suggest that information on features such as voicing are embedded in both LSTM and transformer-based representations. 1 Introduction On the one hand, writing is an essential form of human communication. Writing systems and or- thographies differ across languages and impact our reading behavior. Psycholinguists have extensively studied the effect of orthographic depth, i.e., the transparency of grapheme-to-phoneme mappings, on reading acquisition as well as skilled reading (Seymour et al., 2003). On the other hand, the wide range of cross- linguistic diversity is still a major challenge for nat- ural language processing (NLP) and for the study of language more generally (Mielke et al., 2019; Gutierrez-Vasques and Mijangos, 2020), especially on sub-word levels (Gutierrez-Vasques et al., 2021). This increases the importance of cross-lingual anal- yses of character-level language models (LMs), be- cause anglocentrism in linguistic research is not only prevalent in NLP, but also in (reading and) orthography research (Share, 2008). Character-based language models have gained significant attention in recent years in languages with Latin scripts, since they contain meaningful in- formation on various linguistic levels and enhance the robustness of models. Oh et al. (2021) sug- gest that character LMs provide a more human-like account of sentence processing, which assumes a larger role of morphology, phonotactics, and ortho- graphic complexity than was previously thought. Moreover, including character and sub-character information in LMs for Asian scripts is a standard practice. Despite of this recent attention, work fo- cusing on getting a deeper understanding of char- acter representation is scarce (Kann and Monsalve- Mercado, 2021), in particular regarding the com- parison between languages and different types of scripts. The goal of this work is to improve our under- standing of learned character representations, for better interpretability of the models. Like other neural network based models, character-level LMs can be seen as black-box methods and reveal lim- ited insights about the causes for their predictions (Gilpin et al., 2018). We investigate the information encoded in character embeddings by comparing them to perceptual representations. Such represen- tations …

Content truncated to 500 words