SLING: Sino Linguistic Evaluation of Large Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4606 - 4634 December 7-11, 2022 ©2022 Association for Computational Linguistics SLING: Sino LINGuistic Evaluation of Large Language Models Yixiao Song♢ Kalpesh Krishna♠ Rajesh Bhatt♢ Mohit Iyyer♠ ♢Department of Linguistics, UMass Amherst ♠Manning College of Information and Computer Sciences, UMass Amherst {yixiaosong,bhatt}@umass.edu {kalpesh,miyyer}@cs.umass.edu Abstract To understand what kinds of linguistic knowl- edge are encoded by pretrained Chinese lan- guage models (LMs), we introduce the bench- mark of Sino LINGuistics (SLING), which con- sists of 38K minimal sentence pairs in Man- darin Chinese grouped into 9 high-level lin- guistic phenomena. Each pair demonstrates the acceptability contrast of a specific syntactic or semantic phenomenon (e.g., The keys are lost vs. The keys is lost), and an LM should assign lower perplexity to the acceptable sen- tence. In contrast to the CLiMP dataset (Xiang et al., 2021), which also contains Chinese min- imal pairs and was created by translating the vocabulary of the English BLiMP dataset, the minimal pairs in SLING are derived primarily by applying syntactic and lexical transforma- tions to naturally-occurring, linguist-annotated sentences from the Chinese Treebank 9.0, thus addressing severe issues in CLiMP’s data gener- ation process. We test 18 publicly available pre- trained monolingual (e.g., BERT-base-zh, CPM) and multi-lingual (e.g., mT5, XLM) lan- guage models on SLING. Our experiments show that the average accuracy for LMs is far below human performance (69.7% vs. 97.1%), while BERT-base-zh achieves the highest accuracy (84.8%) of all tested LMs, even much larger ones. Additionally, we find that most LMs have a strong gender and number (sin- gular/plural) bias, and they perform better on local phenomena than hierarchical ones.1 1 Introduction While large-scale pretrained language models (LMs) have achieved considerable downstream suc- cess (Devlin et al., 2019; Xue et al., 2021; Brown et al., 2020, a.o.), it remains challenging to evalu- ate how much linguistic knowledge they have ac- quired. One approach is to design minimal pairs consisting of two sentences that differ only in a 1The SLING data and code can be found https:// github.com/Yixiao-Song/SLING_Data_Code. A: 他们在吃饭了。 (They are already in the process of having a meal.) B: 他们在吃了饭。 (They are already in the process of having finished a meal.) Chinese Speakers (97.1% acc.) Pretrained LMs (69.7% acc.) B A Contribution #1: 38K minimal edit pairs in Chinese across nine syntactic and semantic linguistic phenomena Contribution #2: 18 large Chinese LMs evaluated (monolingual as well as multilingual like mT5, XLM). LMs underperform native speakers (97% vs 70%) Figure 1: An illustration of the SLING dataset. The A sentence is acceptable but B, a minimal edit counterpart of A, is not. LMs see one sentence at a time and are expected to assign a lower (pseudo-)perplexity to the ac- ceptable sentence. Overall, LMs underperform Chinese native speakers on SLING (97% vs 70%), making it an exciting benchmark for future Chinese LM research. critical word or phrase, which renders only one of the sentences acceptable (e.g., The keys are lost vs. The …

Content truncated to 500 words