Composite Attention
method
ID: compositeattention
Related Papers
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models
AbstractIn this paper, we detail the relationship between convolutions and self-attention in natural language tasks. We show that relative position embeddings in self-attention layers are equivalent to recently-proposed dynamic lightweight …...