GPT Perdetry Test: Generating new meanings for new words

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5542–5553 June 6–11, 2021. ©2021 Association for Computational Linguistics 5542 GPT Perdetry Test: Generating new meanings for new words Nikolay Malkin1 Yale University New Haven, CT Sameera Lanka2 Microsoft Redmond, WA Pranav Goel3 University of Maryland College Park, MD Sudha Rao2 Microsoft Research Redmond, WA Nebojsa Jojic2 Microsoft Research Redmond, WA 1kolya.malkin@yale.edu 2{sameera.lanka,sudha.rao,jojic}@microsoft.com 3pgoel1@umd.edu Abstract Human innovation in language, such as invent- ing new words, is a challenge for pretrained language models. We assess the ability of one large model, GPT-3, to process new words and decide on their meaning. We create a set of nonce words and prompt GPT-3 to generate their dictionary definitions. We find GPT-3 produces plausible definitions that align with human judgments. Moreover, GPT-3’s defi- nitions are sometimes preferred to those in- vented by humans, signaling its intriguing abil- ity not just to adapt, but to add to the evolving vocabulary of the English language. 1 Introduction Humans are constantly expanding languages with new words. How are artificial language models, which are increasingly deployed ‘in the wild’, to handle the stream of neologisms that are appearing in slang or on social media (Grieve et al., 2018)? Today’s most advanced language models, includ- ing GPT-3 (Brown et al., 2020), use a subword tokenization of input text, rather than consuming it word by word. This allows them to process words never seen in their training data. For example, the word ‘perdetry’, which has never been used in En- glish, is treated by GPT-3 as a sequence of two tokens (Fig. 1). The subword tokenization algo- rithm is designed for text compression and does not respect the natural morpheme boundaries. We explore GPT-3’s understanding of English at the subword level by prompting it to give defini- tions of nonce words1 (Fig. 1). We find in human 1We use the term ‘nonce word’ for a new word not used in English. It becomes a ‘neologism’ once it acquires a meaning. LSTM butherment (n.) GPT-3 happiness or pride in having big plans and no perseverance per|detry (n.) - an instance of inventing words, esp. as a hobby har|bole|mic (adj.) - tending to babble; talking nonsense sh|out|ze (v.) - to laugh through half-open teeth Figure 1: Above: The neologism generation pipeline. Below: Definitions created by GPT-3 for three nonce words. Prompts bold, subword tokens separated by |. studies that not only does GPT-3 generate realistic, original meanings for new words, but its definitions are sometimes preferred to those invented by hu- mans. This finding sheds light on GPT-3’s ability to adapt to and even extend a changing vocabulary. While we cannot ascertain GPT-3’s exact rea- sons for assigning meanings to nonce words, our results prove that these reasons are not limited to morphology: many neologisms have no clear roots or derivational origin. The meanings of words may be imported by their phonological qualities – more precisely, their orthographic realizations – or by …

Content truncated to 500 words