Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 165–173 July 13-14, 2023 ©2023 Association for Computational Linguistics Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise? Xinzhe Li, Ming Liu, Shang Gao School of IT, Deakin University, Australia {lixinzhe, m.liu,shang.gao}@deakin.edu.au Abstract For Pretrained Language Models (PLMs), their susceptibility to noise has recently been linked to subword segmentation. However, it is un- clear which aspects of segmentation affect their understanding. This study assesses the robust- ness of PLMs against various disrupted segmen- tation caused by noise. An evaluation frame- work for subword segmentation, named Con- trastive Lexical Semantic (CoLeS) probe, is proposed. It provides a systematic categoriza- tion of segmentation corruption under noise and evaluation protocols by generating contrastive datasets with canonical-noisy word pairs. Ex- perimental results indicate that PLMs are un- able to accurately compute word meanings if the noise introduces completely different sub- words, small subword fragments, or a large number of additional subwords, particularly when they are inserted within other subwords. 1 Introduction The capability to understand the meaning of noisy words through character arrangements is a crucial aspect of human cognitive abilities (Rawlinson, 2007). This capability is highly sought after in practical applications such as machine translation and sentiment analysis (Belinkov and Bisk, 2018). However, despite their success in in-distribution test data with standardized word forms, Pretrained Language Models (PLMs), which serve as the back- bone models, tend to perform poorly on rare or noisy words (Kumar et al., 2020; Baron, 2015). These noisy words may be caused by accidental ty- pos (Belinkov and Bisk, 2018) or spelling variants on social media (Ritter et al., 2010). Prior studies show that most subword-based PLMs perform poorly under noise largely due to subword segmentation (Zhuang and Zuccon, 2022), while character-based PLMs show more robustness (El Boukkouri et al., 2020). Examining the impact of subword segmentation factors on PLMs is also crucial for defending against the adversarial attacks that leverage the sensitivity of subword segmenta- tion to noise (Liu et al., 2022). However, rare work has investigated how the subword segmentation from noisy words affects the word meaning. To help address this question, we design and de- velop a contrastive framework (CoLes) to assess the robustness of PLMs in the face of various forms of segmentation corruption. As subword segmen- tation can be influenced by noise in various ways, such as adding extra subwords or losing original subwords, we systematically categorize the ways into four main categories and two additional sub- categories based on three subword sets, as exem- plified in Table 1. Two types of noise models are proposed to effectively generate all the types of corruption except missing corruption, and a con- trastive dataset consisting of noisy and standard word pairs is created. This framework enables us to evaluate the significance of preserved subwords and the impact of subwords added by noise. The experimental results provide the following insights: 1) complete corruption: the PLMs strug- gle to infer …

Content truncated to 500 words