Трансформердің өзіндік назары

AbstractFor text generation from semantic graphs, past neural models encoded input structure via gated convolutions along graph edges. Although these operations provide local context, the distance messages can travel is …...

AbstractThe computing cost of transformer self-attention often necessitates breaking long documents to fit in pretrained models in document ranking tasks. In this paper, we design Query-Directed Sparse attention that induces …...

An attention matrix of a transformer self- attention sublayer can provably be decom- posed into two components and only one of them (effective attention) contributes to the model output. This …...