What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3405–3424 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3405 What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers Boseop Kim∗,1 HyoungSeok Kim∗,1 Sang-Woo Lee∗,1,2 Gichang Lee1 Donghyun Kwak1 Dong Hyeon Jeon3 Sunghyun Park4 Sungju Kim1,3 Seonhoon Kim3 Dongpil Seo1 Heungsub Lee1 Minyoung Jeong1 Sungjae Lee1 Minsub Kim1 Suk Hyun Ko1 Seokhun Kim1 Taeyong Park1 Jinuk Kim1 Soyoung Kang1 Na-Hyeon Ryu1 Kang Min Yoo1,2 Minsuk Chang2 Soobin Suh1,3 Sookyo In1,3 Jinseong Park1,3 Kyungduk Kim1,3 Hiun Kim1 Jisu Jeong1,2 Yong Goo Yeo1 Donghoon Ham1 Dongju Park1 Min Young Lee1 Jaewook Kang1 Inho Kang1,3 Jung-Woo Ha1,2 Woomyoung Park1 Nako Sung1 NAVER CLOVA1 NAVER AI Lab2 NAVER Search3 Search Solutions, Inc.4 Abstract GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less re- ported by the GPT-3 paper, such as a non- English LM, the performances of different sized models, and the effect of recently in- troduced prompt optimization on in-context learning. To achieve this, we introduce Hy- perCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific to- kenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt- based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non- experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering inter- face. Lastly, we demonstrate the potential of our methods with three successful in-house ap- plications. 1 Introduction Due to its remarkable zero-shot and few-shot per- formances, GPT-3’s in-context learning has gained significant attention in the AI community (Brown et al., 2020). In the in-context learning approach, discrete prompts that consist of a natural language task description and few-shot examples control ∗Equal contribution. large-scale language models (LMs) to infer pre- dictions for the target task. Using OpenAI’s GPT-3, studies have proposed methods that can further boost in-context learning performance of GPT-3 (Zhao et al., 2021; Liu et al., 2021a). Recently, prompt-based learning methods have been reported to improve the performances of BERT, GPT-3, and T5 without any parameter updates of the main model (Liu et al., 2021b; Lester et al., 2021; Shin et al., 2020). We consider the three following practical issues of using GPT-3. First, the language composition of the training corpus is heavily skewed towards English with 92.7%. This makes it difficult to ap- ply it to tasks in other languages. We also know little about how to train similar models in another language with different linguistic properties, and where the originally proposed methods will be nat- urally applied and where they might fail. Second, while it is pragmatic and useful to know the ca- pabilities …

Content truncated to 500 words