ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7364–7381 July 9-14, 2023 ©2023 Association for Computational Linguistics ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models Jonas Belouadi jonas.belouadi@uni-bielefeld.de Steffen Eger steffen.eger@uni-bielefeld.de Natural Language Learning Group (NLLG) Faculty of Technology, Bielefeld University nl2g.github.io Abstract State-of-the-art poetry generation systems are often complex. They either consist of task- specific model pipelines, incorporate prior knowledge in the form of manually created con- straints, or both. In contrast, end-to-end models would not suffer from the overhead of having to model prior knowledge and could learn the nuances of poetry from data alone, reducing the degree of human supervision required. In this work, we investigate end-to-end poetry genera- tion conditioned on styles such as rhyme, meter, and alliteration. We identify and address lack of training data and mismatching tokenization algorithms as possible limitations of past at- tempts. In particular, we successfully pre-train ByGPT5, a new token-free decoder-only lan- guage model, and fine-tune it on a large custom corpus of English and German quatrains anno- tated with our styles. We show that ByGPT5 outperforms other models such as mT5, ByT5, GPT-2 and ChatGPT, while also being more parameter efficient and performing favorably compared to humans. In addition, we analyze its runtime performance and demonstrate that it is not prone to memorization. We make our code, models, and datasets publicly available.1 1 Introduction End-to-end fine-tuning of pre-trained language mod- els like GPT-2 (Radford et al., 2019) or T5 (Raf- fel et al., 2020a) on downstream tasks has been an immensely popular training paradigm for text- generation in the last few years (Li et al., 2021). End- to-end models learn to complete a task by directly learning all steps, without intermediary algorithms such as hand-crafted rules or post-processing. This approach has proven to be highly effective on a wide range of problems such as dialog generation (Sun et al., 2022; Yang et al., 2021), summarization (Zhu et al., 2021; Zhong et al., 2021; Huang et al., 2021), and machine translation (Farinha et al., 2022; Tran 1https://github.com/potamides/uniformers The sweet wild strain, the sudden start, A Which shakes the perfumed altar’s flame, B To make its shrine a sacred name, B And sing its praise in every heart. A — ByGPT5 Figure 1: Generated quatrain with ABBA rhyme scheme, high amount of alliterations (green), and iambic meter, i.e., unstressed syllable ( ) follows stressed syllable ( ). et al., 2020). Nevertheless, all these applications have in common that they only concern themselves with the generation of prosaic texts. Generating formal verse poetry on the other hand, with strict constraints on aesthetic style such as rhyme scheme, meter and alliteration, remains a difficult problem. Attempts to employ end-to-end solutions in this con- text have so far been unsuccessful (Wöckener et al., 2021), with some authors even concluding that language models cannot pick up such constraints from data alone (Popescu-Belis et al., 2022). As a consequence, state-of-the-art poetry generation sys- tems rely on …

Content truncated to 500 words