Шаблоны внимания
Related Papers
Revealing the Dark Secrets of BERT
AbstractBERT-based architectures currently give state-of-the-art performance on many NLP tasks, but little is known about the exact mechanisms that contribute to its success. In the current work, we focus on …...
How Far Does BERT Look At: Distance-based Clustering and Analysis of BERT’s Attention
AbstractRecent research on the multi-head attention mechanism, especially that in pre-trained models such as BERT, has shown us heuristics and clues in analyzing various aspects of the mechanism. As most …...
Attention Can Reflect Syntactic Structure (If You Let It)
AbstractSince the popularization of the Transformer as a general-purpose feature encoder for NLP, many studies have attempted to decode linguistic structure from its novel multi-head attention mechanism. However, much of …...
Incorporating Residual and Normalization Layers into Analysis of Masked Language Models
AbstractTransformer architecture has become ubiquitous in the natural language processing field. To interpret the Transformer-based models, their attention patterns have been extensively analyzed. However, the Transformer architecture is not only …...
Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translation
AbstractNeural machine translation (NMT) has become the de-facto standard in real-world machine translation applications. However, NMT models can unpredictably produce severely pathological translations, known as hallucinations, that seriously undermine user …...