Предварительная подготовка языковых моделей
method
ID: pretraininglanguagemodels
Related Papers
Does She Wink or Does She Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models
AbstractRecent progress in pretraining language models on large corpora has resulted in significant performance gains on many NLP tasks. These large models acquire linguistic knowledge during pretraining, which helps to …...
Learning Better Masking for Better Language Model Pre-training
AbstractMasked Language Modeling (MLM) has been widely used as the denoising objective in pre-training language models (PrLMs). Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio …...