Toward the Limitation of Code-Switching in Cross-Lingual Transfer

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5966 - 5971 December 7-11, 2022 ©2022 Association for Computational Linguistics Toward the Limitation of Code-Switching in Cross-Lingual Transfer Yukun Feng1 Feng Li2 Philipp Koehn1 1Johns Hopkins University 2University of Illinois Urbana-Champaign {yfeng55, phi}@jhu.edu, fengl3@illinois.edu Abstract Multilingual pretrained models have shown strong cross-lingual transfer ability. Some works used code-switching sentences, which consist of tokens from multiple languages, to enhance the cross-lingual representation fur- ther, and have shown success in many zero-shot cross-lingual tasks. However, code-switched tokens are likely to cause grammatical inco- herence in newly substituted sentences, and negatively affect the performance on token- sensitive tasks, such as Part-of-Speech (POS) tagging and Named-Entity-Recognition (NER). This paper mitigates the limitation of the code- switching method by not only making the token replacement but considering the similarity be- tween the context and the switched tokens so that the newly substituted sentences are gram- matically consistent during both training and inference. We conduct experiments on cross- lingual POS and NER over 30+ languages, and demonstrate the effectiveness of our method by outperforming the mBERT by 0.95 and original code-switching method by 1.67 on F1 scores. 1 Introduction Recent studies(Devlin et al., 2019; Lample and Conneau, 2019; Conneau et al., 2020a) have shown the success of multilingual corpus pre-training for cross-lingual knowledge transfer. Some works(Qin et al., 2020; Yang et al., 2021) further addressed the effectiveness of code-switching on improving the performance of multilingual models on zero- shot cross-lingual tasks(Conneau et al., 2018; Hu et al., 2020). Though code-switching has shown great potential and strong generalization ability on the semantic representation, the newly switched se- quence fails to consider the token-level coherence. Specifically, a code-switched sentence consists of tokens from various languages, and such words are likely to cause grammatical incoherence, resulting in an inconsistent context space for the newly sub- stituted sentence. Also, code-switched tokens are How do menschen look at and अनुभव 艺术 ? How do people look at and experience art ? How do people look at and experience art ? How do Menschen look at and अनुभव 艺术 ? (b): Ours (a): Code-Switched Training Sentence SCONJ AUX NOUN VERB ADP CCPNJ VERB NOUN PUNCT Ground Truth Labels: Training: Inference: 人们如何看待和体验艺术? लोग कला को कै से देखते ह� और अनुभव German Wie betrachten und erleben Menschen Kunst? Hindi Chinese mBERT mBERT mBERT mBERT mBERT Figure 1: Overview of the training and inference for POS in code-switching and our settings. conditioned on the original context, for which, dur- ing the optimization phase, the embedding of sub- stituted tokens is greatly affected. However, such updates do not consider the token-level dependency in their own language and will likely cause inconsis- tent embedding space during inference. Therefore, even though code-switching benefits most of the sentence-level cross-lingual tasks, it still limits the performance of token-level prediction. To address such issues, we use an alignment strat- egy to reduce the gap between original context and code-switched tokens during training. Specifically, code-switched tokens are …

Content truncated to 500 words