LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1803–1813 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1803 LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation Gustavo Aguilar, Sudipta Kar, Thamar Solorio University of Houston Department of Computer Science {gaguilaralas, skar3, tsolorio}@uh.edu Abstract Recent trends in NLP research have raised an interest in linguistic code-switching (CS); modern approaches have been proposed to solve a wide range of NLP tasks on multiple language pairs. Unfortunately, these proposed methods are hardly generalizable to different code-switched languages. In addition, it is unclear whether a model architecture is applicable for a different task while still being compatible with the code-switching setting. This is mainly because of the lack of a centralized benchmark and the sparse corpora that researchers employ based on their specific needs and interests. To facilitate research in this direction, we propose a centralized benchmark for Linguistic Code-switching Evaluation (LinCE) that combines ten corpora covering four different code-switched language pairs (i.e., Spanish-English, Nepali-English, Hindi-English, and Modern Standard Arabic-Egyptian Arabic) and four tasks (i.e., language identification, named entity recognition, part-of-speech tagging, and sentiment analysis). As part of the benchmark centralization effort, we provide an online platform at ritual.uh.edu/lince, where researchers can submit their results while comparing with others in real-time. In addition, we provide the scores of different popular models, including LSTM, ELMo, and multilingual BERT so that the NLP community can compare against state-of-the-art systems. LinCE is a continuous effort, and we will expand it with more low-resource languages and tasks. Keywords: code-switching, code-mixing, multilingualism, benchmark 1. Introduction Linguistic code-switching1 (CS) is the multilingual phe- nomenon that happens when speakers alternate languages within the same sentence or utterance. During the last decade, the CS phenomenon has attracted more research interest from the NLP community. Many researchers have proposed novel methods to handle code-switched data, showing improvements on core NLP tasks such as language identification (LID), named entity recognition (NER), and part-of-speech (POS) tagging. However, many of these ap- proaches are usually evaluated on a few language pairs and a specific domain, and it is not clear whether these mod- els are exclusive to such scenarios or they can generalize to other tasks, domains, and language pairs. Moreover, research in code-switching currently has a slow process of comparison in which researchers have to repli- cate previous methods to report scores on different datasets. Furthermore, choosing the best-published model for bench- marking purposes is not an easy task either. These problems exist mainly because 1) there is no official benchmark for general code-switching evaluation that allows direct com- parisons across multiple tasks, and 2) methods are usually not comprehensively evaluated across datasets with differ- ent language pairs. To overcome these problems, we propose a centralized Linguistic Code-switching Evaluation (LinCE) bench- mark. We have consolidated a benchmark from preexisting corpora considering the following aspects: 1) multiple lan- guage pairs from high- and low-resource languages with a reasonable range of code-mixing indexes (CMI) (Gamb¨ack and Das, …

Content truncated to 500 words