Прямые сети
Related Papers
On the Computational Power of Transformers and Its Implications in Sequence Modeling
AbstractTransformers are being used extensively across several sequence modeling tasks. Significant research effort has been devoted to experimentally probe the inner workings of Transformers. However, our conceptual and theoretical understanding …...
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
AbstractRecent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear. …...
Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models’ Memories
AbstractPre-trained language models (PLMs) demonstrate excellent abilities to understand texts in the generic domain while struggling in a specific domain. Although continued pre-training on a large domain-specific corpus is effective, …...
Transformer-based Models for Long-Form Document Matching: Challenges and Empirical Analysis
AbstractRecent advances in the area of long document matching have primarily focused on using transformer-based models for long document encoding and matching. There are two primary challenges associated with these …...