KoBEST: Korean Balanced Evaluation of Significant Tasks

Proceedings of the 29th International Conference on Computational Linguistics, pages 3697–3708 October 12–17, 2022. 3697 KOBEST: Korean Balanced Evaluation of Significant Tasks Myeongjun Jang1∗ Dohyung Kim2∗ Deuk Sin Kwon2 Eric Davis2 1Department of Computer Science, University of Oxford, UK myeongjun.jang@cs.ox.ac.uk 2Language Super Intelligence Labs, SK Telecom, South Korea {bruno, brian.k, eric.davis}@sktair.com Abstract A well-formulated benchmark plays a critical role in spurring advancements in the natural language processing (NLP) field, as it allows objective and precise evaluation of diverse models. As modern language models (LMs) have become more elaborate and sophisti- cated, more difficult benchmarks that require linguistic knowledge and reasoning have been proposed. However, most of these benchmarks only support English, and great effort is nec- essary to construct benchmarks for other low resource languages. To this end, we propose a new benchmark named Korean balanced evaluation of significant tasks (KoBEST), which consists of five Korean-language down- stream tasks. Professional Korean linguists de- signed the tasks that require advanced Korean linguistic knowledge. Moreover, our data is purely annotated by humans and thoroughly reviewed to guarantee high data quality. We also provide baseline models and human per- formance results. Our dataset is available on the Huggingface 1. 1 Introduction The NLP field is now facing unprecedented rapid development. A major factor propelling the progress is the existence of unified benchmark datasets like GLUE (Wang et al., 2018), which are designed to assess models’ language understand- ing capabilities. Such benchmark datasets, enabled modern pre-trained language models (PLMs), such as BERT (Devlin et al., 2019) and GPT-2 and GPT-3 (Radford et al., 2019; Brown et al., 2020), to be assessed in objective and multifaceted manners. The success of GLUE also lead to sim- ilar benchmark datasets in a variety of other lan- guages, such as French (Le et al., 2020), Ko- rean (Park et al., 2021), Chinese (Xu et al., 2020) and Indonesian (Wilie et al., 2020). *These authors contributed equally to this work 1https://huggingface.co/datasets/skt/kobest_v1 However, many recent studies reveal that the outstanding performance of PLMs on such bench- mark datasets seems plausible but not probable. These studies have found that datasets may con- tain many spurious artefacts, and the performance of PLMs is enhanced by excessive usage of said artefacts (Habernal et al., 2018; Niven and Kao, 2019; McCoy et al., 2019; Bender and Koller, 2020). Another line of work observed that many PLMs, which showed promising results in GLUE, fall short of expectations for more difficult tasks that require linguistic knowledge (Bhatt et al., 2021) or logical reasoning (Tian et al., 2021). As a result, the importance of well-designed evalua- tion datasets with higher difficulty-level has been highlighted, and new datasets, such as CHECK- LIST (Ribeiro et al., 2020), and LOGICNLI (Tian et al., 2021), have been proposed. Most of them only support specific languages like English, and it requires large efforts to build higher difficulty- level language evaluation suits for other low re- source languages, however. When it comes to the Korean language, two benchmarks are widely used: Korean-NLI & …

Content truncated to 500 words