Optimizing Deeper Transformers on Small Datasets

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2089–2102 August 1–6, 2021. ©2021 Association for Computational Linguistics 2089 Optimizing Deeper Transformers on Small Datasets Peng Xu1, Dhruv Kumar∗1,2, Wei Yang1, Wenjie Zi1, Keyi Tang1, Chenyang Huang∗1,5, Jackie Chi Kit Cheung1,3,4, Simon J.D. Prince1, Yanshuai Cao1 1Borealis AI 2University of Waterloo 3McGill University 4Canada CIFAR Chair, Mila 5University of Alberta {peng.z.xu, wei.yang, wenjie.zi, keyi.tang, simon.prince, yanshuai.cao}@borealisai.com dhruv.kumar@uwaterloo.ca, chuang8@ualberta.ca, jcheung@cs.mcgill.ca Abstract It is a common belief that training deep trans- formers from scratch requires large datasets. Consequently, for small datasets, people usu- ally use shallow and simple additional lay- ers on top of pre-trained models during fine- tuning. This work shows that this does not al- ways need to be the case: with proper initial- ization and optimization, the benefits of very deep transformers can carry over to challeng- ing tasks with small datasets, including Text- to-SQL semantic parsing and logical reading comprehension. In particular, we success- fully train 48 layers of transformers, com- prising 24 fine-tuned layers from pre-trained RoBERTa and 24 relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state-of-the-art performance on the challeng- ing cross-domain Text-to-SQL parsing bench- mark Spider1. We achieve this by deriving a novel Data-dependent Transformer Fixed- update initialization scheme (DT-Fixup), in- spired by the prior T-Fixup work (Huang et al., 2020). Further error analysis shows that in- creasing depth can help improve generaliza- tion on small datasets for hard cases that re- quire reasoning and structural understanding. 1 Introduction In recent years, large-scale pre-trained language models (Radford et al., 2019; Devlin et al., 2018; Liu et al., 2019b) trained with transform- ers (Vaswani et al., 2017) have become standard building blocks of modern NLP systems to help improve generalization when task-specific annota- tions are limited. In practice, it has been found that deeper transformers generally yield better re- sults with sufficient training data (Lan et al., 2019), ∗Work done while the author was an intern in Borealis AI. 1The code to reproduce our results can be found in: https://github.com/BorealisAI/DT-Fixup especially on tasks involving reasoning and struc- tural understanding. This suggests that additional transformer layers should be employed in conjunc- tion with pre-trained models, instead of simple and shallow neural components, such as a classifier head, currently used by models of many NLP tasks. However, the common belief in the literature is that training deep transformers from scratch requires large datasets, and few attempts have been made on small datasets, to the best of our knowledge. One implication is that although extra transformer lay- ers on top of pre-trained models should help with more challenging problems in principle, it does not work in practice due to limited training data. We show that after resolving several optimization issues with the method proposed in this work, it is possible to train very deep transformers with improved generalization even on small datasets. One advantage of pre-trained …

Content truncated to 500 words