Character-Aware Models Improve Visual Text Rendering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 16270–16297 July 9-14, 2023 ©2023 Association for Computational Linguistics Character-Aware Models Improve Visual Text Rendering Rosanne Liu∗, Dan Garrette∗, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, Noah Constant∗ Google Research {rosanneliu, dhgarrette, nconstant}@google.com Abstract Current image generation models struggle to reliably produce well-formed visual text. In this paper, we investigate a key contribut- ing factor: popular text-to-image models lack character-level input features, making it much harder to predict a word’s visual makeup as a series of glyphs. To quantify this effect, we conduct a series of experiments compar- ing character-aware vs. character-blind text en- coders. In the text-only domain, we find that character-aware models provide large gains on a novel spelling task (WikiSpell). Applying our learnings to the visual domain, we train a suite of image generation models, and show that character-aware variants outperform their character-blind counterparts across a range of novel text rendering tasks (our DrawText benchmark). Our models set a much higher state-of-the-art on visual spelling, with 30+ point accuracy gains over competitors on rare words, despite training on far fewer examples. 1 Introduction Over the last year, image generation models have made impressive quality gains (Rombach et al., 2021; Ramesh et al., 2022; Saharia et al., 2022; Yu et al., 2022). While many practical use cases are al- ready within reach, rendering visual text in images remains a challenge. Ramesh et al. (2022) observe that DALL·E-2 “struggles at producing coherent text,” and the latest release of Stable Diffusion lists “cannot render legible text” as a known limitation.1 In this paper, we seek to understand and improve the ability of image generation models to render high-quality visual text. To do so, we first investi- gate the spelling ability of text encoders in isolation. We find that despite their popularity, character- blind text encoders—which receive no direct signal as to the character-level makeup of their inputs— have limited spelling ability. Building on Itzhak ∗Equal contribution. 1https://huggingface.co/stabilityai/stable-diffusion-2-1 and Levy (2022), we test the spelling ability of text encoders across scales, architectures, input rep- resentations, languages, and tuning methods. We document for the first time the miraculous ability of character-blind models to induce robust spelling knowledge (>99% accuracy) through web pretrain- ing, but show that this does not generalize well beyond English, and is only achieved at scales over 100B parameters, making it infeasible for most applications. We find that character-aware text encoders, on the other hand, are able to achieve robust spelling ability at far smaller scales. Applying these findings to image generation, we train a range of character-aware text-to-image mod- els and demonstrate that they significantly outper- form character-blind models on text rendering. For purely character-level models, this improved text rendering comes at a cost—decreasing image-text alignment for prompts that don’t involve visual text. To alleviate this, we propose combining character- level and token-level input representations, and find that this delivers the best of both …

Content truncated to 500 words