RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 921 - 931 May 22-27, 2022 c⃝2022 Association for Computational Linguistics RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining Hui Su1∗, Weiwei Shi1∗, Xiaoyu Shen2∗ Xiao Zhou1, Tuo Ji1, Jiarui Fang1, and Jie Zhou1 1Pattern Recognition Center, Wechat AI, Tencent Inc, China 2Saarland Informatics Campus aaronsu@tencent.com Abstract Large-scale pretrained language models have achieved SOTA results on NLP tasks. How- ever, they have been shown vulnerable to ad- versarial attacks especially for logographic lan- guages like Chinese. In this work, we propose ROCBERT: a pretrained Chinese Bert that is robust to various forms of adversarial attacks like word perturbation, synonyms, typos, etc. It is pretrained with the contrastive learning ob- jective which maximizes the label consistency under different synthesized adversarial exam- ples. The model takes as input multimodal information including the semantic, phonetic and visual features. We show all these features are important to the model robustness since the attack can be performed in all the three forms. Across 5 Chinese NLU tasks, ROCBERT out- performs strong baselines under three black- box adversarial algorithms without sacrificing the performance on clean testset. It also per- forms the best in the toxic content detection task under human-made attacks. 1 Introduction Large-scale pretrained models, by finetuning on sufficient annotated data, have been able to ap- proach or even surpass human performance on many benchmark testsets (Peters et al., 2018; Rad- ford et al.; Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020). However, even pretrained with huge amounts of text, the models are still vul- nerable under adversarial attacks like synonyms, word deletion/swapping, misspelling, etc (Li et al., 2019a; Jin et al., 2020; Sun et al., 2020a; Eger and Benz, 2020). These adversarial examples occur frequently in the real-world scenario and can be made either naturally (e.g., typos) or maliciously (e.g., to avoid auto detection of toxic content) 1. ∗Equal contribution. 1The concept of adversarial examples is quite wide. In this paper, we focus on adversarial examples that do NOT change the original semantics (Mozes et al., 2021) Attacker Text phonetic 克(kè)比的精神值得永远学习 visual 科此的精神值得永远学习 character split 禾斗匕匕的精神值得永远学习 synonym 我(wˇo)科(k¯e)的精神值得永远学习 synonym + phonetic 蜗(w¯o)壳(ké)的精神值得永远学习 to pinyin kebi的精神值得永远学习 to pinyin + unicode keb1的精神值得学习永远 swap 科比的精神值得永远习学 insertion 科比的精神九值得永远学习 deletion 科比的神值得永远学习 Original: 科(k¯e)比(bˇı)的精神值得永远学习 Translation: Kobe’s spirit is worth studying forever. Table 1: Examples of various attackers. Contents in the brackets are corresponding pinyins of Chinese characters. The lack of robustness with them can easily lead to large performance drop when testing in the noisy real-world traffic. The issue is particularly out- standing for logographic languages like Chinese since the attack can be either with the glyph charac- ter, pinyin (the romanized phonetic representations) or a combination of them (Wang et al., 2020; Li et al., 2020d; Zhang et al., 2020; Nuo et al., 2020). We show some examples in Table 1. The word “ 科 比(Kobe)” can be replaced with synonyms, phonet- ically or visually similar words. The …

Content truncated to 500 words