Native-like Expression Identification by Contrasting Native and Proficient Second Language Speakers

Proceedings of the 28th International Conference on Computational Linguistics, pages 592–603 Barcelona, Spain (Online), December 8-13, 2020 592 Arabizi Language Models for Sentiment Analysis Souhir Gahbiche and Guillaume Gadek Airbus, Advanced Information Processing, France Ga´etan Baert and Alexandre Pauchet Litis, INSA Rouen Normandie alexandre.pauchet@insa-rouen.fr Abstract Arabizi is a written form of spoken Arabic, relying on Latin characters and digits. It is informal and does not follow any conventional rules, raising many NLP challenges. In particular, Arabizi has recently emerged as the Arabic language in online social networks, becoming of great interest for opinion mining and sentiment analysis. Unfortunately, only few Arabizi resources exist and state-of-the-art language models such as BERT do not consider Arabizi. In this work, we construct and release two datasets: (i) LAD, a corpus of 7.7M tweets written in Arabizi and (ii) SALAD, a subset of LAD, manually annotated for sentiment analysis. Then, a BERT architecture is pre-trained on LAD, in order to create and distribute an Arabizi language model called BAERT. We show that a language model (BAERT) pre-trained on a large corpus (LAD) in the same language (Arabizi) as that of the fine-tuning dataset (SALAD), outperforms a state-of-the-art multi-lingual pretrained model (multilingual BERT) on a sentiment analysis task. 1 Introduction Nowadays, Online Social Networks (OSNs) are highly popular means of communication. With pictures, videos, urls, mentions and of course comments, people share ideas, opinions and sentiments on Twitter1, Facebook2, LinkedIn3 and many other service providers. OSN posts are of great interest for marketing and reputation management, enabling companies to collect feedback about brands and products. Au- tomatic systems facilitating these large-scale tasks, including sentiment classifiers, have progressively reached promising accuracy (Abbasi et al., 2008; Catal and Nangir, 2017; Giatsoglou et al., 2017). In parallel, the NLP field has experienced several scientific developments until the recent language models. Efficient representation of texts with vectors is difficult and this challenge has seen excellent work: bag-of-words with tf*idf (Robertson et al., 1995), Word2Vec (Mikolov et al., 2013), GloVe (Pen- nington et al., 2014), FastText (Joulin et al., 2016), ELMo (Peters et al., 2018), and BERT (Devlin et al., 2018) for instance. Each of these scientific steps has successively outperformed the previous ones on a variety of NLP tasks including sentiment analysis, with a focus on English texts. Moreover, some approaches such as BERT propose pre-trained language models, including a multi-lingual version. The particularity of texts post on OSN is that they frequently mix made-up terms and inventive spelling with more classic syntax. Twitter, for instance, contains invented words, original spelling and typing er- rors (Maynard and Funk, 2011), but also includes special entities such as hashtags and user mentions that often appear as labels without any syntactic role (Gadek et al., 2017). Even state-of-the-art architectures such as BERT usually suffer from the particularities of OSN languages, as illustrated on a task of hate detection in tweets (Gertner et al., 2019). Moreover, OSN posts are also the opportunity for code-switching, i.e. alternating between two or more alphabets and/or languages …

Content truncated to 500 words