Is GPT-3 a Good Data Annotator?

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11173–11195 July 9-14, 2023 ©2023 Association for Computational Linguistics Is GPT-3 a Good Data Annotator? Bosheng Ding∗1,2 Chengwei Qin∗1 Linlin Liu†1,2 Yew Ken Chia2 Boyang Li1 Shafiq Joty1 Lidong Bing‡2 1Nanyang Technological University, Singapore 2DAMO Academy, Alibaba Group {bosheng001, chengwei003, linlin001, boyang.li, srjoty}@ntu.edu.sg {bosheng.ding, yewken.chia, l.bing}@alibaba-inc.com Abstract Data annotation is the process of labeling data that could be used to train machine learning models. Having high-quality annotation is cru- cial, as it allows the model to learn the rela- tionship between the input data and the desired output. GPT-3, a large-scale language model developed by OpenAI, has demonstrated im- pressive zero- and few-shot performance on a wide range of NLP tasks. It is therefore natural to wonder whether it can be used to effectively annotate data for NLP tasks. In this paper, we evaluate the performance of GPT-3 as a data annotator by comparing it with traditional data annotation methods and analyzing its output on a range of tasks. Through this analysis, we aim to provide insight into the potential of GPT-3 as a general-purpose data annotator in NLP 1. 1 Introduction The democratization of artificial intelligence (AI) (Garvey, 2018; Rubeis et al., 2022) aims to pro- vide access to AI technologies to all members of society, including individuals, small- and medium- sized enterprises (SMEs), academic research labs, and nonprofit organizations. Achieving this goal is crucial for the promotion of innovation, economic growth, and fairness and equality. As typical AI models are usually data-hungry, one significant ob- stacle of AI democratization is the preparation of well-annotated data for training AI models. Specifically, supervised learning critically de- pends on sufficient training data with accurate an- notation, but data annotation can be a costly en- deavor, particularly for small-scale companies and organizations (Bunte et al., 2021). The cost of data ∗Equal contribution, order decided by coin flip. Bosheng Ding are under the Joint PhD Program between Alibaba and Nanyang Technological University. †Work done while under the Joint PhD Program between Alibaba and Nanyang Technological University. ‡Correspondent Author. 1Our code is available at https://github.com/ DAMO-NLP-SG/LLM-Data-Annotator. annotation typically includes the labor costs associ- ated with the labeling process, as well as the time and resources required to hire, train and manage annotators. Additionally, there may be costs asso- ciated with the annotation tools and infrastructure needed to support the annotation process. Individ- uals or small-scale organizations may not have re- sources to annotate sufficient training data, thereby are unable to reap the benefits of contemporary AI technologies. Although the development of pre- trained language models such as BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), GPT-2 (Radford et al., 2019) and RoBERTa (Liu et al., 2019) eases the data-hungry issue to some extent, data annotation remains an unavoidable challenge for supervised model training. GPT-3 (Brown et al., 2020; Ouyang et al., 2022)2 is a powerful large language model developed by OpenAI. Evaluations show that GPT-3 …

Content truncated to 500 words