JBLiMP: Japanese Benchmark of Linguistic Minimal Pairs

Findings of the Association for Computational Linguistics: EACL 2023, pages 1581–1594 May 2-6, 2023 ©2023 Association for Computational Linguistics JBLiMP: Japanese Benchmark of Linguistic Minimal Pairs Taiga Someya and Yohei Oseki The University of Tokyo {taiga98-0809,oseki}@g.ecc.u-tokyo.ac.jp Abstract In this paper, we introduce JBLiMP (Japanese Benchmark of Linguistic Minimal Pairs), a novel dataset for targeted syntactic evaluations of language models in Japanese. JBLiMP consists of 331 minimal pairs, which are cre- ated based on acceptability judgments extracted from journal articles in theoretical linguistics. These minimal pairs are grouped into 11 cate- gories, each covering a different linguistic phe- nomenon. JBLiMP is unique in that it com- bines two important features independently ob- served in existing datasets: (i) coverage of com- plex linguistic phenomena (cf. CoLA) and (ii) presentation of sentences as minimal pairs (cf. BLiMP). In addition, JBLiMP is the first dataset for targeted syntactic evaluations of lan- guage models in Japanese, thus allowing the comparison of syntactic knowledge of language models across different languages. We then evaluate the syntactic knowledge of several lan- guage models on JBLiMP: GPT-2, LSTM, and n-gram language models. The results demon- strated that all the architectures achieved com- parable overall accuracies around 75%. Error analyses by linguistic phenomenon further re- vealed that these language models successfully captured local dependencies like nominal struc- tures, but not long-distance dependencies such as verbal agreement and binding. 1 Introduction The past few years have seen a remarkable suc- cess of neural language models, and some language models based on Transformer (Vaswani et al., 2017) have achieved the state-of-the-art performance in various natural language processing (NLP) tasks (Wang et al., 2018, 2019). In fact, recent neural lan- guage models are extremely successful in solving a variety of downstream tasks, but it remains to be understood how well these neural language mod- els understand the syntax of natural languages. In order to address this question, some studies investi- gated the syntactic knowledge of language models with a specially designed dataset for targeted syn- tactic evaluations (e.g., Linzen et al., 2016; Marvin and Linzen, 2018; Wilcox et al., 2018; Gulordava et al., 2018; Futrell et al., 2019; Chaves, 2020). However, most of these studies have focused on En- glish and other European languages, and only few studies extended this investigation to non-European languages (Gulordava et al., 2018; Ravfogel et al., 2018). Importantly for the purpose here, even fewer studies have dealt with a wide variety of linguistic phenomena in non-English languages (Xiang et al., 2021; Trotta et al., 2021). In this paper, we introduce JBLiMP (Japanese Benchmark of Linguistic Minimal Pairs), a novel dataset for targeted syntactic evaluations of lan- guage models in Japanese.1 JBLiMP consists of 331 minimal pairs, which are created based on ac- ceptability judgments extracted from journal ar- ticles in theoretical linguistics. These minimal pairs are grouped into 11 categories, each cover- ing a different linguistic phenomenon. JBLiMP is unique in that it successfully combines two important features independently observed in ex- isting datasets: (i) coverage of complex …

Content truncated to 500 words