Cross-modal Memory Networks for Radiology Report Generation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 536–549 August 1–6, 2021. ©2021 Association for Computational Linguistics 536 Cascaded Head-colliding Attention Lin Zheng♠♦ Zhiyong Wu♠ Lingpeng Kong♠♣ ♠Department of Computer Science, The University of Hong Kong ♦School of Data and Computer Science, Sun Yat-sen University ♣Shanghai Artificial Intelligence Laboratory zhenglin6@mail2.sysu.edu.cn,{zywu,lpk}@cs.hku.hk Abstract Transformers have advanced the field of natu- ral language processing (NLP) in many ways. At the heart of the Transformer architecture is the multi-head attention (MHA) mechanism which models pairwise interactions between the elements of the sequence. Despite its mas- sive success, the current framework ignores in- teractions among different heads, leading to the problem that many of the heads are redun- dant in practice, which underutilizes the ca- pacity of the model. To improve parameter efficiency, we re-formulate the MHA as a la- tent variable model from a probabilistic per- spective. We present cascaded head-colliding attention (CODA) which explicitly models the interactions between attention heads through a hierarchical variational distribution. We con- duct extensive experiments and demonstrate that CODA outperforms the transformer base- line, by 0.6 perplexity on Wikitext-103 in language modeling, and by 0.6 BLEU on WMT14 EN-DE in machine translation, due to its improvements on the parameter efficiency.1 1 Introduction Transformers (Vaswani et al., 2017) have advanced the field of natural language processing (NLP) on a variety of important tasks, including language mod- eling (Dai et al., 2019; Baevski and Auli, 2019), language understanding (Devlin et al., 2019; Yang et al., 2019b), and machine translation (Vaswani et al., 2017; Dehghani et al., 2019; Liu et al., 2020). It has also found its place in computer vi- sion (Dosovitskiy et al., 2020), and in intelligent agents (Vinyals et al., 2019) where sequence mod- eling plays a key role as well. The cornerstone of the transformer architecture is the multi-head at- tention (MHA) mechanism which models pairwise interactions between the elements of the sequence. 1Our implementation is publicly available at https:// github.com/LZhengisme/CODA. An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility func- tion of the query with the corresponding key. A multi-head attention (MHA) mechanism extends the idea through performing multiple separately pa- rameterized attention functions acting in parallel to contextualize the input representations. Their out- puts are then gathered by an affine transformation, allowing the model to jointly attend to information from different representation subspaces at different positions. Despite its massive success, the current frame- work ignores the interactions among different heads, leading to the problem that many of the heads are redundant in practice (i.e., attending to the same regions of the sequence), which under- utilizes the capacity of the model (Voita et al., 2019; Michel et al., …

Content truncated to 500 words