Keeping an Eye on Context: Attention Allocation over Input Partitions in Referring Expression Generation

Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023), pages 20–27 September 12, 2023 ©2023 Association for Computational Linguistics Keeping an Eye on Context: Attention Allocation over Input Partitions in Referring Expression Generation Simeon Schüz and Sina Zarrieß Bielefeld University {simeon.schuez,sina.zarriess}@uni-bielefeld.de Abstract In Referring Expression Generation, model in- puts are often composed of different represen- tations, including the visual properties of the intended referent, its relative position and size, and the visual context. Yet, the extent to which this information influences the generation pro- cess of black-box neural models is largely un- clear. We investigate the relative weighting of target, location, and context information in the attention components of a Transformer-based generation model.1 Our results show a general target bias, which, however, depends on the content of the generated expressions, pointing to interesting directions for future research. 1 Introduction Context is crucial in multimodal language genera- tion tasks such as Referring Expression Generation (REG), as descriptions for visible entities not only depend on their own appearance but also on their surroundings (e.g. Yu et al. 2016). For REG, this is especially evident, as the same expression can unambiguously describe an object in one context but be misleading in others (Schüz et al., 2023). To this end, it has become a common practice to provide neural generation models for multimodal REG not only with visual representations for the target itself but also with information about its lo- cation and size and the visual context it appears in (see Figure 1). However, due to their black- box nature, it is not entirely clear to which extend state-of-the-art neural REG models take all of these representations into consideration. While ablation studies show how context information contribute to model performance (Yu et al., 2016; Zarrieß and Schlangen, 2018), they provide limited insight into how it is processed and to what extend it is relevant for e.g. lexical decisions. 1Code and models for this project are available at: https://github.com/clause-bielefeld/REG-Input-Partitions Figure 1: Input for our REG model. Input vectors are concatenations of visual (Vt) and location (Loct) fea- tures for targets and visual context features (Vc). We examine the relative attention weights of each partition. Similar questions arise in other vision & lan- guage (V&L) tasks such as image captioning, where recent work has looked into analyzing the in- ternal attention mechanisms of generation models (e.g. Ilinykh and Dobnik 2021, 2022). However, as the respective models usually take global images as input, analyses are mostly concerned with atten- tion distribution within those representations rather than across different parts of the input. For REG, some authors use attention heatmaps as a method of model introspection (Tanaka et al., 2019; Liu et al., 2020; Sun et al., 2022), but without analyzing the patterns in detail. As a first step for deeper investigations of how contextual information is processed in REG mod- els, we quantify how attention is allocated over the partitioned inputs in a simple REG model. In more detail, we …

Content truncated to 500 words