CINO: A Chinese Minority Pre-trained Language Model

Proceedings of the 29th International Conference on Computational Linguistics, pages 3937–3949 October 12–17, 2022. 3937 CINO: A Chinese Minority Pre-trained Language Model Ziqing Yang†, Zihang Xu†, Yiming Cui‡†∗, Baoxin Wang†, Min Lin, Dayong Wu†, Zhigang Chen†§ †State Key Laboratory of Cognitive Intelligence, iFLYTEK Research, Beijing, China ‡Research Center for SCIR, Harbin Institute of Technology, Harbin, China §Jilin Kexun Information Technology Co., Ltd., Changchun, China †{zqyang5,zhxu13,ymcui,bxwang2,dywu2,zgchen}@iflytek.com ‡ymcui@ir.hit.edu.cn Abstract Multilingual pre-trained language models have shown impressive performance on cross- lingual tasks. It greatly facilitates the ap- plications of natural language processing on low-resource languages. However, there are still some languages that the current multi- lingual models do not perform well on. In this paper, we propose CINO (Chinese Mi- nority Pre-trained Language Model), a mul- tilingual pre-trained language model for Chi- nese minority languages. It covers Standard Chinese, Yue Chinese, and six other ethnic minority languages. To evaluate the cross- lingual ability of the multilingual model on ethnic minority languages, we collect doc- uments from Wikipedia and news websites, and construct two text classification datasets, WCM (Wiki-Chinese-Minority) and CMNews (Chinese-Minority-News). We show that CINO notably outperforms the baselines on various classification tasks. The CINO model and the datasets are publicly available at http: //cino.hfl-rc.com. 1 Introduction The multilingual pre-trained language model (MPLM) is known for its ability to understand multiple languages, and its surprising zero-shot cross-lingual ability (Wu and Dredze, 2019). The zero-shot cross-lingual transfer ability enables the MPLM to be applied on the target languages with limited or even no annotated data by fine-tuning the MPLM on the source language with rich annotated data. MPLMs greatly facilitate transferring the cur- rent NLP technologies to low-resource languages and reduce the cost of developing NLP applications for low-resource languages. The existing public MPLMs such as mBERT (Devlin et al., 2019), XLM (Conneau and Lam- ple, 2019) and XLM-R (Conneau et al., 2020) can ∗Email corresponding. handle 100 languages, but there are still some chal- lenges on low-resource languages understanding: • The size of pre-training corpora of some low- resource languages is small compared to the high-resource languages. This bias towards high- resource languages may harm the performance on low-resource languages. • There are thousands of living languages in the world, but many languages have not been covered in the existing MPLMs, especially indigenous or ethnic minority languages. For example, Tibetan, a language spoken mainly by Tibetans around Tibetan Plateau, is absent from the CC-100 cor- pus. Therefore, the XLM-R tokenizer can not tokenize Tibetan scripts correctly, and XLM-R is not good at understanding Tibetan texts. Recently, more advanced MPLMs have been pro- posed, such as ERNIE-M (Ouyang et al., 2021), VECO (Luo et al., 2021) and Unicoder (Huang et al., 2019). These models focus on multilingual training objectives, such as leveraging parallel sen- tences to improve the alignment between different languages, and have improved notably over XLM- R. However, these models have not paid attention to the low-resource languages, so the problem re- mains unsolved. For the above reasons, it is necessary to …

Content truncated to 500 words