Трансформерные кодировщики

AbstractThis paper presents a new dataset, B-SHARP, that can be used to develop NLP models for the detection of Mild Cognitive Impairment (MCI) known as an early sign of Alzheimer’s …...

We study the possibilities of building a non-autoregressive speech-to-text translation model using connectionist temporal classifi- cation (CTC), and use CTC-based automatic speech recognition as an auxiliary task to im- prove …...

With the ever-increasing complexity of neu- ral language models, practitioners have turned to methods for understanding the predictions of these models. One of the most well- adopted approaches for model …...

AbstractMulti-task learning with transformer encoders (MTL) has emerged as a powerful technique to improve performance on closely-related tasks for both accuracy and efficiency while a question still remains whether or …...

AbstractIn this paper, we investigate improvements to the GEC sequence tagging architecture with a focus on ensembling of recent cutting-edge Transformer-based encoders in Large configurations. We encourage ensembling models by …...

AbstractThe eXtreme Multi-label text Classification (XMC) problem concerns finding most relevant labels for an input text instance from a large label set. However, the XMC setup faces two challenges: (1) …...

AbstractGenerative models have recently started to outperform extractive models in Open Domain Question Answering, largely by leveraging their decoder to attend over multiple encoded passages and combining their information. However, …...

AbstractThis paper analyzes three formal models of Transformer encoders that differ in the form of their self-attention mechanism: unique hard attention (UHAT); generalized unique hard attention (GUHAT), which generalizes UHAT; …...

AbstractTransformer encoders contextualize token representations by attending to all other tokens at each layer, leading to quadratic increase in compute effort with the input length. In practice, however, the input …...