LAMB

This paper introduces STEM and LAMB, em- beddings trained for stems and lemmata in- stead of for surface forms. For morpholog- ically rich languages, they perform signif- icantly better than …...

AbstractAdaptive gradient methods, such as Adam and LAMB, have demonstrated excellent performance in the training of large language models. Nevertheless, the need for adaptivity requires maintaining second-moment estimates of the …...