Scheduled Masking Approaches
method
ID: scheduledmaskingapproaches
Related Papers
Learning Better Masking for Better Language Model Pre-training
AbstractMasked Language Modeling (MLM) has been widely used as the denoising objective in pre-training language models (PrLMs). Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio …...