JGLUE: Japanese General Language Understanding Evaluation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2957–2966 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2957 JGLUE: Japanese General Language Understanding Evaluation 1Kentaro Kurihara, 1Daisuke Kawahara, 2Tomohide Shibata 1Waseda University, 2Yahoo Japan Corporation {kkurihara@akane., dkw@}waseda.jp, tomshiba@yahoo-corp.jp Abstract To develop high-performance natural language understanding (NLU) models, it is necessary to have a benchmark to evaluate and analyze NLU ability from various perspectives. While the English NLU benchmark, GLUE (Wang et al., 2018), has been the forerunner, benchmarks are now being released for languages other than English, such as CLUE (Xu et al., 2020) for Chinese and FLUE (Le et al., 2020) for French; but there is no such benchmark for Japanese. We build a Japanese NLU benchmark, JGLUE, from scratch without translation to measure the general NLU ability in Japanese. We hope that JGLUE will facilitate NLU research in Japanese. Keywords: GLUE, Japanese, NLU benchmark, text classification, sentence pair classification, QA 1. Introduction To develop high-performance natural language under- standing (NLU) models, it is necessary to have a bench- mark (a set of datasets) to evaluate and analyze NLU ability from various perspectives. In the case of En- glish, the GLUE (General Language Understanding Evaluation) benchmark (Wang et al., 2018) is publicly available. Once an NLU model that can achieve a cer- tain level of high score on GLUE is developed, a more difficult benchmark, such as SuperGLUE (Wang et al., 2019), is released, creating a virtuous cycle of bench- mark construction and NLU model development. Along with the trend of active NLU studies in En- glish, benchmarks for languages other than English have been constructed, including CLUE (Xu et al., 2020) for Chinese, FLUE (Le et al., 2020) for French, and KLUE (Park et al., 2021) for Korean. Although there are many studies on Japanese, which is the 13th most spoken language in the world as of 2021, there is no benchmark such as GLUE. Japanese is lin- guistically different from English and other languages in the following aspects. • The Japanese alphabet includes hiragana, katakana, Chinese characters, and the Latin alphabet. • There are no spaces between words. • The word order is relatively free. Due to these differences, findings on English datasets are not necessarily applicable to Japanese. Given this situation, there is an urgent need to develop a bench- mark for Japanese NLU. Although individual Japanese datasets, such as JSNLI (Yoshikoshi et al., 2020) and JSICK (Yanaka and Mineshima, 2021), have been con- structed, their construction methods involve mainly machine translation or manual translation from English datasets. With either of these translation methods, the unnaturalness of a translated text and the cultural/social Task Dataset Train Dev Test Text MARC-ja 187,528 5,654 5,639 Classification JCoLA — — — Sentence Pair JSTS 12,451 1,457 1,589 Classification JNLI 20,073 2,434 2,508 QA JSQuAD 63,870 4,475 4,470 JCommonsenseQA 9,012 1,119 1,118 Table 1: JGLUE overview. discrepancy between an original language (mostly En- glish) and a target language (Japanese in …

Content truncated to 500 words