Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1375–1391 May 2-6, 2023 ©2023 Association for Computational Linguistics Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining Asa Cooper Stickland⋆¶†, Sailik Sengupta⋆‡, Jason Krone¶‡, He He‡♢, Saab Mansour‡ †University of Edinburgh, ‡�WS AI Labs, ♢New York University a.cooper.stickland@ed.ac.uk,{sailiks,saabm,hehea}@amazon.com Abstract Advances in neural modeling have achieved state-of-the-art (SOTA) results on public nat- ural language processing (NLP) benchmarks, at times surpassing human performance. How- ever, there is a gap between public benchmarks and real-world applications where noise, such as typographical or grammatical mistakes, is abundant and can result in degraded perfor- mance. Unfortunately, works which evaluate the robustness of neural models on noisy data and propose improvements, are limited to the English language. Upon analyzing noise in dif- ferent languages, we observe that noise types vary greatly across languages. Thus, exist- ing investigations do not generalize trivially to multilingual settings. To benchmark the per- formance of pretrained multilingual language models, we construct noisy datasets covering five languages and four NLP tasks and observe a clear gap in the performance between clean and noisy data in the zero-shot cross-lingual set- ting. After investigating several ways to boost the robustness of multilingual models in this setting, we propose Robust Contrastive Pre- training (RCP). RCP combines data augmenta- tion with a contrastive loss term at the pretrain- ing stage and achieves large improvements on noisy (& original test data) across two sentence- level (+3.2%) and two sequence-labeling (+10 F1-score) multilingual classification tasks. 1 Introduction Recently, multilingual pre-trained language mod- els like mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020) and various others (Chi et al., 2021; Xue et al., 2021; Chi et al., 2022) have improved multilingual language understand- ing by pretraining large Transformer models on web-scale corpora (such as Wikipedia, Common- Crawl). These models achieve state-of-the-art per- formance on cross-lingual transfer and many mul- tilingual NLP tasks (Wu and Dredze, 2019; Pires ⋆ Equal Contribution, ¶ Work done while at Amazon. et al., 2019). However, a real-world system will encounter real-world noise, such as linguistic vari- ations and common errors observed in textual data, that are often absent from benchmark datasets. While prior works focused on this issue of ro- bustness in monolingual settings (Peng et al., 2021; Sengupta et al., 2021; Tan et al., 2020), investiga- tion has been scarce for multilingual settings. In this paper, we study the effect of realistic noise in multilingual settings and propose methods to boost the robustness of multilingual language mod- els across four NLP tasks: Intent Classification (IC), Slot Labeling (SL), Named Entity Recogni- tion (NER) and Natural Language Inference (NLI). Due to the lack of multilingual noisy evaluation data, we synthesize benchmarks by mining noise from publicly available corpora and injecting them into the test sets associated with each of the four tasks. We conduct human validation to ensure that this noised data is indeed realistic (see examples from …

Content truncated to 500 words