Предварительное обучение языковой модели
Related Papers
Linguistic Knowledge and Transferability of Contextual Representations
AbstractContextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language. To shed …...
A Neural Local Coherence Analysis Model for Clarity Text Scoring
AbstractLocal coherence relation between two phrases/sentences such as cause-effect and contrast gives a strong influence of whether a text is well-structured or not. This paper follows the assumption and presents …...
Enhancing Clinical BERT Embedding using a Biomedical Knowledge Base
AbstractDomain knowledge is important for building Natural Language Processing (NLP) systems for low-resource settings, such as in the clinical domain. In this paper, a novel joint training method is introduced …...
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models
AbstractIn this paper, we detail the relationship between convolutions and self-attention in natural language tasks. We show that relative position embeddings in self-attention layers are equivalent to recently-proposed dynamic lightweight …...
Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
AbstractRecent progress in language model pre-training has led to important improvements in Named Entity Recognition (NER). Nonetheless, this progress has been mainly tested in well-formatted documents such as news, Wikipedia, …...