What the DAAM: Interpreting Stable Diffusion Using Cross Attention

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5644–5659 July 9-14, 2023 ©2023 Association for Computational Linguistics What the DAAM: Interpreting Stable Diffusion Using Cross Attention Raphael Tang,∗1 Linqing Liu,∗2 Akshat Pandey,1 Zhiying Jiang,3 Gefei Yang,1 Karun Kumar,1 Pontus Stenetorp,2 Jimmy Lin,3 Ferhan Ture1 1Comcast Applied AI 2University College London 3University of Waterloo 1{raphael_tang,akshat_pandey,gefei_yang,karun_kumar,ferhan_ture}@comcast.com 2{linqing.liu,p.stenetorp}@cs.ucl.ac.uk 3{zhiying.jiang,jimmylin}@uwaterloo.ca Abstract Diffusion models are a milestone in text-to- image generation, but they remain poorly un- derstood, lacking interpretability analyses. In this paper, we perform a text–image attribution analysis on Stable Diffusion, a recently open- sourced model. To produce attribution maps, we upscale and aggregate cross-attention maps in the denoising module, naming our method DAAM. We validate it by testing its segmenta- tion ability on nouns, as well as its generalized attribution quality on all parts of speech, rated by humans. On two generated datasets, we at- tain a competitive 58.8–64.8 mIoU on noun seg- mentation and fair to good mean opinion scores (3.4–4.2) on generalized attribution. Then, we apply DAAM to study the role of syntax in the pixel space across head–dependent heat map in- teraction patterns for ten common dependency relations. We show that, for some relations, the head map consistently subsumes the dependent, while the opposite is true for others. Finally, we study several semantic phenomena, focusing on feature entanglement; we find that the pres- ence of cohyponyms worsens generation qual- ity by 9%, and descriptive adjectives attend too broadly. We are the first to interpret large diffu- sion models from a visuolinguistic perspective, which enables future research. Our code is at https://github.com/castorini/daam. 1 Introduction Diffusion models trained on billions of captioned images represent state-of-the-art text-to-image gen- eration (Yang et al., 2022), with some achieving photorealism, such as Google’s Imagen (Saharia et al., 2022) and OpenAI’s DALL-E 2 (Ramesh et al., 2022). However, despite their quality and popularity, the dynamics of their image synthesis remain undercharacterized. Citing ethics, corpora- tions have restricted the general public from using the models and their weights, preventing effective ∗Equal contribution. Figure 1: The original synthesized image and three DAAM maps for “monkey,” “hat,” and “walking,” from the prompt, “monkey with hat walking.” analysis. To overcome this barrier, Stability AI recently open-sourced Stable Diffusion (Rombach et al., 2022), a 1.1 billion-parameter latent diffusion model pretrained and fine-tuned on the LAION 5- billion image dataset (Schuhmann et al., 2022). We probe Stable Diffusion to provide insight into large diffusion models. Focusing on text–image at- tribution, our central question is, “How does an input word influence parts of a generated image?” To this, we propose to produce 2D attribution maps for each word by combining cross-attention maps in the model. A related work in prompt-guided editing from Hertz et al. (2022) conjectures that per-head cross attention relates words to areas in Imagen-generated images, but they fall short of constructing global per-word attribution maps. We name our method diffusion attentive attribution maps, or “DAAM;” see Figure 1 for an example. To …

Content truncated to 500 words