Learning the Legibility of Visual Text Perturbations

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3260–3273 May 2-6, 2023 ©2023 Association for Computational Linguistics Learning the Legibility of Visual Text Perturbations Dev Seth1 Rickard Stureborg1 Danish Pruthi∗ Bhuwan Dhingra1 1 Duke University {ds447, rs541, bd149}@duke.edu Abstract Many adversarial attacks in NLP perturb in- puts to produce visually similar strings (‘ergo’ → ‘εrgo’) which are legible to humans but de- grade model performance. Although preserv- ing legibility is a necessary condition for text perturbation, little work has been done to sys- tematically characterize it; instead, legibility is typically loosely enforced via intuitions around the nature and extent of perturbations. Par- ticularly, it is unclear to what extent can in- puts be perturbed while preserving legibility, or how to quantify the legibility of a perturbed string. In this work, we address this gap by learning models that predict the legibility of a perturbed string, and rank candidate perturba- tions based on their legibility. To do so, we collect and release LEGIT, a human-annotated dataset comprising the legibility of visually per- turbed text. Using this dataset, we build both text- and vision-based models which achieve up to 0.91 F1 score in predicting whether an input is legible, and an accuracy of 0.86 in predict- ing which of two given perturbations is more legible. Additionally, we discover that legible perturbations from the LEGIT dataset are more effective at lowering the performance of NLP models than best-known attack strategies, sug- gesting that current models may be vulnerable to a broad range of perturbations beyond what is captured by existing visual attacks.1 1 Introduction To manage the increasing demand for content moderation—e.g., detecting spam or toxic/hateful content on online platforms—organizations have turned to machine learning solutions. In response, users often resort to manipulating text to evade de- tection, removal, or search. For instance, hateful comments often comprise of visually similar char- acters to avoid automatic filtering (Le et al., 2022). ∗ Work done while at Carnegie Mellon University. 1Data, code, and models are available at https://github. com/dvsth/learning-legibility-2023. “Small De” Script: Cyrillic U+0434 “Th with strikethrough” Script: Latin Extended U+1D7A “Double Integral” Script: Math Symbol U+222C “Small Yu” Script: Cyrillic U+044E Anonymous @anonymous @anonymous Anonymous @anonymous @anonymous @anonymous Figure 1: Visual attacks in the wild. Examples of Twitter users manipulating their tweets to evade the platform’s ‘sensitive content’ detection algorithms. Since people read text visually, the manipulated content can still be easily understood and harm its target audience. These attacks started with sim- ple ASCII substitutions like he11o (colloquially referred to as “leetspeak"), but have evolved into complex manipulations utilizing characters from different Unicode scripts (Flamand, 2008; Ray- mond, 1996). Figure 1 shows two such examples. Unlike computer vision where there is an estab- lished notion of what constitutes an imperceptible perturbation (typically defined via the ℓ∞ distance), most perturbations in text are perceptible. However, as long as the perceptible manipulations remain legible, the message could have its intended effect. The legibility of a text is …

Content truncated to 500 words