Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 460–469 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 460 Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer Siddhant Garg∗ † Amazon Alexa AI Search Manhattan Beach, CA, USA sidgarg@amazon.com Rohit Kumar Sharma∗ † Microsoft Seattle, WA, USA rsharma@cs.wisc.edu Yingyu Liang University of Wisconsin-Madison Madison, WI, USA yliang@cs.wisc.edu Abstract Fine-tuning (FT) pre-trained sentence embed- ding models on small datasets has been shown to have limitations. In this paper we show that concatenating the embeddings from the pre- trained model with those from a simple sen- tence embedding model trained only on the tar- get data, can improve over the performance of FT for few-sample tasks. To this end, a linear classifier is trained on the combined embed- dings, either by freezing the embedding model weights or training the classifier and embed- ding models end-to-end. We perform evalua- tion on seven small datasets from NLP tasks and show that our approach with end-to-end training outperforms FT with negligible com- putational overhead. Further, we also show that sophisticated combination techniques like CCA and KCCA do not work as well in prac- tice as concatenation. We provide theoretical analysis to explain this empirical observation. 1 Introduction Fine-tuning (FT) powerful pre-trained sentence em- bedding models like BERT (Devlin et al., 2018) has recently become the de-facto standard for down- stream NLP tasks. Typically, FT entails jointly learning a classifier over the pre-trained model while tuning the weights of the latter. While FT has been shown to improve performance on tasks like GLUE (Wang et al., 2018) having large datasets (QQP, MNLI, QNLI), similar trends have not been observed on small datasets, where one would ex- pect the maximum benefits of using a pre-trained model. Several works (Phang et al., 2018; Garg et al., 2019; Dodge et al., 2020; Lee et al., 2020) have demonstrated that FT with a few target domain samples is unstable with high variance, thereby of- ten leading to sub-par gains. Furthermore, this ∗Equal contribution by authors †Work completed at the University of Wisconsin-Madison issue has also been well documented in practice 1. Learning with low resources has recently be- come an active research area in NLP, and arguably one of the most interesting scenarios for which pre-trained models are useful (e.g., (Cherry et al., 2019)). Many practical applications have small datasets (e.g., in social science, medical studies, etc), which are different from large-scale academic benchmarks having hundreds of thousands of train- ing samples (e.g, DBpedia (Lehmann et al., 2015), Sogou News (Wang et al., 2008), etc). This neces- sitates effective transfer learning approaches using pre-trained sentence embedding models for few- sample tasks. In this work, we show that concatenating sen- tence embeddings from a pre-trained model and those from a smaller model trained solely on the target data, can improve over the performance of FT. Specifically, we first learn a simple sentence embedding model …

Content truncated to 500 words