CLiMP: A Benchmark for Chinese Language Model Evaluation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2784–2790 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2784 CLiMP: A Benchmark for Chinese Language Model Evaluation Beilei Xiang,1 Changbing Yang,1 Yu Li,1 Alex Warstadt2 and Katharina Kann1 1University of Colorado Boulder, 2New York University {beilei.xiang, changbing.yang, yuli9309}@colorado.edu warstadt@nyu.edu katharina.kann@colorado.edu Abstract Linguistically informed analyses of language models (LMs) contribute to the understand- ing and improvement of these models. Here, we introduce the corpus of Chinese linguistic minimal pairs (CLiMP), which can be used to investigate what knowledge Chinese LMs acquire. CLiMP consists of sets of 1,000 minimal pairs (MPs) for 16 syntactic con- trasts in Mandarin, covering 9 major Mandarin linguistic phenomena. The MPs are semi- automatically generated, and human agree- ment with the labels in CLiMP is 95.8%. We evaluate 11 different LMs on CLiMP, covering n-grams, LSTMs, and Chinese BERT. We find that classifier–noun agreement and verb com- plement selection are the phenomena that mod- els generally perform best at. However, mod- els struggle the most with the bˇa construction, binding, and filler-gap dependencies. Over- all, Chinese BERT achieves an 81.8% average accuracy, while the performances of LSTMs and 5-grams are only moderately above chance level. 1 Introduction Language models (LMs) are crucial parts of natu- ral language processing (NLP) systems for a large variety of tasks, including summarization, machine translation, and dialog generation. More recently, they have become popular in the form of pretrained models,1 which are then fine-tuned on downstream tasks and often obtain state-of-the-art performance (Peters et al., 2018; Devlin et al., 2019; Conneau et al., 2020). However, which linguistic phenom- ena language models can or cannot learn is still poorly understood for many languages. Resources for the syntactic evaluation of LMs, such as BLiMP (Warstadt et al., 2020) have focused 1Throughout this paper, we adopt a broad definition of LMs, which includes language representation models which have been trained on a masked language modeling objective. mainly on English, and non-English resources currently only cover a small set of phenomena (Mueller et al., 2020; Gulordava et al., 2018; Rav- fogel et al., 2018). In order to spur the analysis and subsequent improvement of LMs in Chinese, we introduce the corpus of Chinese linguistic minimal pairs (CLiMP), which can be used to evaluate LMs’ knowledge of Chinese grammar. CLiMP consists of 16 individual datasets that are semi-automatically generated from grammar tem- plates. Each set—or paradigm—contains 1,000 minimal pairs (MPs). Together, they cover 9 core linguistic phenomena in Chinese. Human agreement on this corpus is 95.8%, confirming that CLiMP represents robust contrasts in Chinese grammar. High performance on CLiMP thus im- plies high correlation with human acceptability judgments across these phenomena. We use CLiMP to study Chinese BERT (Devlin et al., 2019),2 6 LSTM (Hochreiter and Schmidhu- ber, 1997) LMs, and 4 5-gram LMs. We evaluate for each MP whether the LM assigns a higher prob- ability to the grammatical or the ungrammatical sentence. Our results show that …

Content truncated to 500 words