UniXcoder: Unified Cross-Modal Pre-training for Code Representation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 46 - 56 May 22-27, 2022 c⃝2022 Association for Computational Linguistics AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, Reut Tsarfaty Department of Computer Science, Bar Ilan University, Ramat-Gan, Israel {aseker00,elronbandel,dbareket,brusli1, shakedgreenfeld,reut.tsarfaty}@gmail.com Abstract Large Pre-trained Language Models (PLMs) have become ubiquitous in the development of language understanding technology and lie at the heart of many artificial intelligence ad- vances. While advances reported for English using PLMs are unprecedented, reported ad- vances using PLMs for Hebrew are few and far between. The problem is twofold. First, so far, Hebrew resources for training large lan- guage models are not of the same magnitude as their English counterparts. Second, most benchmarks available to evaluate progress in Hebrew NLP require morphological bound- aries which are not available in the output of PLMs. In this work we remedy both aspects. We present AlephBERT, a large PLM for Mod- ern Hebrew, trained on larger vocabulary and a larger dataset than any Hebrew PLM before. Moreover, we introduce a novel neural architec- ture that recovers the morphological segments encoded in contextualized embedding vectors. Based on this new morphological component we offer an evaluation suite consisting of mul- tiple tasks and benchmarks that cover sentence- level, word-level and sub-word level analyses. On all tasks, AlephBERT obtains state-of-the- art results beyond contemporary Hebrew state- of-the-art models. We make our AlephBERT model, the morphological extraction compo- nent, and the Hebrew evaluation suite publicly available, for future investigations and evalua- tions of Hebrew PLMs. 1 Introduction Contextualized word representations provided by models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), GPT3 (Brown et al., 2020), T5 (Raffel et al., 2020) and more, were shown in recent years to be a critical component for obtaining state-of-the-art performance on a wide range of Natural Language Processing (NLP) tasks, from surface syntactic tasks as tagging and parsing, to downstream semantic tasks as question answer- ing, information extraction and text summarization. While advances reported for English using such models are unprecedented, previously reported re- sults using PLMs in Modern Hebrew are far from satisfactory. Specifically, the BERT-based Hebrew section of multilingual-BERT (Devlin et al., 2019) (henceforth, mBERT), did not provide a similar boost in performance as observed by the English section of mBERT. In fact, for several reported tasks, the results of the mBERT model are on a par with pre-neural models or neural models based on non-contextual embeddings (Tsarfaty et al., 2020; Klein and Tsarfaty, 2020). An additional Hebrew BERT-based model, HeBERT (Chriqui and Yahav, 2021), has been recently released, yet without em- pirical evidence of performance improvements on key components of the Hebrew NLP pipeline. The challenge of developing PLMs for morphologically-rich and medium-resourced lan- guages such as Modern Hebrew is twofold. First, contextualized word representations are obtained by pre-training a large language model on massive quantities of …

Content truncated to 500 words