Syntax-augmented Multilingual BERT for Cross-lingual Transfer

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4538–4554 August 1–6, 2021. ©2021 Association for Computational Linguistics 4538 Syntax-augmented Multilingual BERT for Cross-lingual Transfer Wasi Uddin Ahmad†∗, Haoran Li‡, Kai-Wei Chang†, Yashar Mehdad‡ †University of California, Los Angeles, ‡Facebook AI †{wasiahmad,kwchang}@cs.ucla.edu, ‡{aimeeli,mehdad}@fb.com Abstract In recent years, we have seen a colossal effort in pre-training multilingual text encoders us- ing large-scale corpora in many languages to facilitate cross-lingual transfer learning. How- ever, due to typological differences across lan- guages, the cross-lingual transfer is challeng- ing. Nevertheless, language syntax, e.g., syn- tactic dependencies, can bridge the typologi- cal gap. Previous works have shown that pre- trained multilingual encoders, such as mBERT (Devlin et al., 2019), capture language syn- tax, helping cross-lingual transfer. This work shows that explicitly providing language syn- tax and training mBERT using an auxiliary objective to encode the universal dependency tree structure helps cross-lingual transfer. We perform rigorous experiments on four NLP tasks, including text classification, question an- swering, named entity recognition, and task- oriented semantic parsing. The experiment re- sults show that syntax-augmented mBERT im- proves cross-lingual transfer on popular bench- marks, such as PAWS-X and MLQA, by 1.4 and 1.6 points on average across all languages. In the generalized transfer setting, the perfor- mance boosted significantly, with 3.9 and 3.1 points on average in PAWS-X and MLQA. 1 Introduction Cross-lingual transfer reduces the requirement of labeled data to perform natural language process- ing (NLP) in a target language, and thus has the ability to avail NLP applications in low-resource languages. However, transferring across languages is challenging because of linguistic differences at levels of morphology, syntax, and semantics. For example, word order difference is one of the cru- cial factors that impact cross-lingual transfer (Ah- mad et al., 2019). The two sentences in English and Hindi, as shown in Figure 1 have the same ∗Work done during internship at Facebook AI. Figure 1: Two parallel sentences in English and Hindi from XNLI (Conneau et al., 2018) dataset. The words highlighted with the same color have the same mean- ing. Although the sentences have a different word or- der, their syntactic dependency structure is similar. meaning but a different word order (while English has an SVO (Subject-Verb-Object) order, Hindi follows SOV). However, the sentences have a similar dependency structure, and the constituent words have similar part-of-speech tags. Presum- ably, language syntax can help to bridge the typo- logical differences across languages. In recent years, we have seen a colossal effort to pre-train Transformer encoder (Vaswani et al., 2017) on large-scale unlabeled text data in one or many languages. Multilingual encoders, such as mBERT (Devlin et al., 2019) or XLM-R (Con- neau et al., 2020) map text sequences into a shared multilingual space by jointly pre-training in many languages. This allows us to transfer the multilin- gual encoders across languages and have found effective for many NLP applications, including text classification (Bowman et al., 2015; …

Content truncated to 500 words