Көпбасты назар
Related Papers
On The Alignment Problem In Multi-Head Attention-Based Neural Machine Translation
AbstractThis work investigates the alignment problem in state-of-the-art multi-head attention models based on the transformer architecture. We demonstrate that alignment extraction in transformer models can be improved by augmenting an …...
Information Aggregation for Multi-Head Attention with Routing-by-Agreement
AbstractMulti-head attention is appealing for its ability to jointly extract different types of information from multiple representation subspaces. Concerning the information aggregation, a common practice is to use a concatenation …...
A Mixture of h - 1 Heads is Better than h Heads
AbstractMulti-head attentive neural architectures have achieved state-of-the-art results on a variety of natural language processing tasks. Evidence has shown that they are overparameterized; attention heads can be pruned without significant …...
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
AbstractTransformer-based models have brought a radical change to neural machine translation. A key feature of the Transformer architecture is the so-called multi-head attention mechanism, which allows the model to focus …...
Investigation of Transformer-based Latent Attention Models for Neural Machine Translation
Current neural translation networks are based on an effective attention mechanism that can be considered as an implicit probabilistic notion of alignment. Such architectures do not guarantee a high quality …...
Learning Source Phrase Representations for Neural Machine Translation
AbstractThe Transformer translation model (Vaswani et al., 2017) based on a multi-head attention mechanism can be computed effectively in parallel and has significantly pushed forward the performance of Neural Machine …...
Mongolian Questions Classification Based on Mulit-Head Attention
AbstractQuestion classification is a crucial subtask in question answering system. Mongolian is a kind of few resource language. It lacks public labeled corpus. And the complex morphological structure of Mongolian …...
Multi-Unit Transformers for Neural Machine Translation
AbstractTransformer models achieve remarkable success in Neural Machine Translation. Many efforts have been devoted to deepening the Transformer by stacking several units (i.e., a combination of Multihead Attentions and FFN) …...
Pheonix at SemEval-2020 Task 5: Masking the Labels Lubricates Models for Sequence Labeling
AbstractThis paper presents the deep-learning model that is submitted to the SemEval-2020 Task 5 competition: “Detecting Counterfactuals”. We participated in both Subtask1 and Subtask2. The model proposed in this paper …...
Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference
AbstractThe neural attention mechanism plays an important role in many natural language processing applications. In particular, multi-head attention extends single-head attention by allowing a model to jointly attend information from …...
Roles and Utilization of Attention Heads in Transformer-based Neural Language Models
AbstractSentence encoders based on the transformer architecture have shown promising results on various natural language tasks. The main impetus lies in the pre-trained neural language models that capture long-range dependencies …...
Seeing Both the Forest and the Trees: Multi-head Attention for Joint Classification on Different Compositional Levels
AbstractIn natural languages, words are used in association to construct sentences. It is not words in isolation, but the appropriate use of hierarchical structures that conveys the meaning of the …...
Sign Language Recognition with Transformer Networks
AbstractSign languages are complex languages. Research into them is ongoing, supported by large video corpora of which only small parts are annotated. Sign language recognition can be used to speed …...
Tencent submission for WMT20 Quality Estimation Shared Task
AbstractThis paper presents Tencent’s submission to the WMT20 Quality Estimation (QE) Shared Task: Sentence-Level Post-editing Effort for English-Chinese in Task 2. Our system ensembles two architectures, XLM-based and Transformer-based Predictor-Estimator …...
Cascaded Head-colliding Attention
AbstractTransformers have advanced the field of natural language processing (NLP) on a variety of important tasks. At the cornerstone of the Transformer architecture is the multi-head attention (MHA) mechanism which …...
Differentiable Subset Pruning of Transformer Heads
AbstractMulti-head attention, a collection of several attention mechanisms that independently attend to different parts of the input, is the key ingredient in the Transformer. Recent work has shown, however, that …...
Multi-level Emotion Cause Analysis by Multi-head Attention Based Multi-task Learning
AbstractEmotion cause analysis (ECA) aims to identify the potential causes behind certain emotions intext. Lots of ECA models have been designed to extract the emotion cause at the clause level. …...
NSYSU-MITLab團隊於福爾摩沙語音辨識競賽2020之語音辨識系統 (NSYSU-MITLab Speech Recognition System for Formosa Speech Recognition Challenge 2020)
In this paper, we describe the system team NSYSU-MITLab implemented for Formosa Speech Recognition Challenge 2020. We use the Transformer architecture 國立中山大學資訊工程學系 Department of Computer Science and Information Engineering …...
TA-MAMC at SemEval-2021 Task 4: Task-adaptive Pretraining and Multi-head Attention for Abstract Meaning Reading Comprehension
AbstractThis paper describes our system used in the SemEval-2021 Task4 Reading Comprehension of Abstract Meaning, achieving 1st for subtask 1 and 2nd for subtask 2 on the leaderboard. We propose …...
Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy
AbstractSimultaneous machine translation (SiMT) generates translation before reading the entire source sentence and hence it has to trade off between translation quality and latency. To fulfill the requirements of different …...
Alleviating the Inequality of Attention Heads for Neural Machine Translation
AbstractRecent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. …...
Doctor Recommendation in Online Health Forums via Expertise Learning
AbstractHuge volumes of patient queries are daily generated on online health forums, rendering manual doctor allocation a labor-intensive task. To better help patients, this paper studies a novel task of …...
Measuring the Mixing of Contextual Information in the Transformer
AbstractThe Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model. Additionally, recent works have …...
Finding the Pillars of Strength for Multi-Head Attention
AbstractRecent studies have revealed some issues of Multi-Head Attention (MHA), e.g., redundancy and over-parameterization. Specifically, the heads of MHA were originally designed to attend to information from different representation subspaces, …...
Transforming Visual Scene Graphs to Image Captions
AbstractWe propose to TransForm Scene Graphs into more descriptive Captions (TFSGC). In TFSGC, we apply multi-head attention (MHA) to design the Graph Neural Network (GNN) for embedding scene graphs. After …...