An Empirical Study of Pre-trained Transformers for Arabic Information Extraction

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4727–4734, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4727 An Empirical Study of Pre-trained Transformers for Arabic Information Extraction Wuwei Lan1, Yang Chen2, Wei Xu2, Alan Ritter2 1 Department of Computer Science and Engineering, Ohio State University 2 School of Interactive Computing, Georgia Institute of Technology lan.105@osu.edu {yang.chen, wei.xu, alan.ritter}@cc.gatech.edu Abstract Multilingual pre-trained Transformers, such as mBERT (Devlin et al., 2019) and XLM- RoBERTa (Conneau et al., 2020a), have been shown to enable the effective cross-lingual zero-shot transfer. However, their perfor- mance on Arabic information extraction (IE) tasks is not very well studied. In this pa- per, we pre-train a customized bilingual BERT, dubbed GigaBERT, that is designed specifi- cally for Arabic NLP and English-to-Arabic zero-shot transfer learning. We study Giga- BERT’s effectiveness on zero-short transfer across four IE tasks: named entity recognition, part-of-speech tagging, argument role labeling, and relation extraction. Our best model signifi- cantly outperforms mBERT, XLM-RoBERTa, and AraBERT (Antoun et al., 2020) in both the supervised and zero-shot transfer set- tings. We have made our pre-trained models publicly available at https://github.com/ lanwuwei/GigaBERT. 1 Introduction Fine-tuning pre-trained Transformer models (De- vlin et al., 2019; Liu et al., 2019; Yang et al., 2019) has recently achieved state-of-the-art results on a wide range of NLP tasks where supervised train- ing data is available. When trained on multilingual corpora, BERT-based models have demonstrated the ability to learn multilingual representations that support zero-shot cross-lingual transfer learning surprisingly effectively (Wu and Dredze, 2019; Pires et al., 2019; Lample and Conneau, 2019). Without access to any parallel text or target lan- guage annotations, multilingual BERT (mBERT; Devlin et al., 2019) even supports cross-lingual transfer for language pairs that are written in differ- ent scripts, for example, English-to-Arabic. How- ever, the transfer learning performance still lags far behind where supervised data is available in the target language. In this paper, we explore to what extent it is possible to improve performance in the zero-shot scenario by building a customized bilin- gual BERT for English and Arabic, a particularly challenging language pair for cross-lingual transfer learning. We present GigaBERT, a customized BERT for English-to-Arabic cross-lingual transfer that is trained on newswire text in the Gigaword corpus (Graff et al., 2003; Parker et al., 2009) in addi- tion to Wikipedia and web crawl data. We sys- tematically compare our pre-trained models of dif- ferent configurations against the mBERT (Devlin et al., 2019) and XLM-RoBERTa (XLM-R; Con- neau et al., 2020a). By using a customized vocabu- lary and code-switched data specifically created for English-to-Arabic transfer learning, our GiagBERT outperforms mBERT and XLM-Rbase (both support more than 100 languages) on a range of IE tasks, including named entity recognition, part-of-speech tagging, argument role labeling, and relation extrac- tion. Further performance gains are demonstrated by augmenting the pre-training corpus with synthet- ically generated code-switched data. This demon- strates the usefulness of anchor points for zero-shot cross-lingual transfer learning. GigaBERT also per- forms well when annotated Arabic …

Content truncated to 500 words