Aligning Multilingual Embeddings for Improved Code-switched Natural Language Understanding

Proceedings of the 29th International Conference on Computational Linguistics, pages 4268–4273 October 12–17, 2022. 4268 Aligning Multilingual Embeddings for Improved Code-switched Natural Language Understanding Barah Fazili IIT Bombay barah@cse.iitb.ac.in Preethi Jyothi IIT Bombay pjyothi@cse.iitb.ac.in Abstract Multilingual pretrained models, while effec- tive on monolingual data, need additional train- ing to work well with code-switched text. In this work, we present a novel idea of train- ing multilingual models with alignment objec- tives using parallel text so as to explicitly align word representations with the same underlying semantics across languages. Such an explicit alignment step has a positive downstream ef- fect and improves performance on multiple code-switched NLP tasks. We explore two alignment strategies and report improvements of up to 7.32%, 0.76% and 1.9% on Hindi- English Sentiment Analysis, Named Entity Recognition and Question Answering tasks compared to a competitive baseline model. 1 Introduction Large pretrained multilingual models have enabled cross-lingual transfer on a number of downstream natural language understanding (NLU) tasks. Apart from serving as a good starting point to train mod- els for tasks in low-resource languages, multilin- gual models (Devlin et al., 2018) (Conneau et al., 2019) have also been used to achieve zero-shot cross-lingual transfer on target languages with no task-specific labeled data. However, compared to monolingual inputs, the effectiveness of multilin- gual models on code-switched inputs—i.e., inputs with two or more languages appearing within or across sentences in a conversation—has not been explored enough. In this work, we aim at explicitly modifying rep- resentations from pretrained multilingual models to be more amenable to code-switched inputs. We do this with the help of parallel text in the two com- ponent languages and alignment objectives that explicitly encourage representations to be better aligned across the two languages. We conjecture that modifying multilingual embeddings to be bet- ter aligned across the two languages will help the model deal better with tokens switching languages within a code-switched sentence. We start with a pretrained multilingual BERT (mBERT) baseline model (Devlin et al., 2018) and design two align- ment objectives to be used with parallel text to align the multilingual embeddings. This “aligned" mBERT model is then further fine-tuned with small amounts of code-switched labeled data in the target task. We find such an aligned model to be more accurate on multiple downstream tasks involving code-switched inputs. The two main highlights of this work can be summarized as follows: • We propose two alignment-based objectives to be used with mBERT and parallel text in En- glish and Hindi. The aligned models are fine- tuned and further evaluated on code-switched Hindi-English NER, SA and QA tasks. Com- pared to the baseline mBERT, we obtain clear improvements on all three downstream tasks. • We investigate how our model behaves in the following two settings: 1) Using a bilingual lexicon instead of parallel text 2) Using Ro- manized Hindi instead of the native Devana- gari script for Hindi. We also present visualizations that clearly show that the alignment objective helps bring representa- tions for aligned words …

Content truncated to 500 words