Data Augmentation for Hypernymy Detection

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 91–104 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 91 BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin David R. Cheriton School of Computer Science, University of Waterloo Vector Institute for Artificial Intelligence {ji.xin,r33tang,yaoliang.yu,jimmylin}@uwaterloo.ca Abstract The slow speed of BERT has motivated much research on accelerating its inference, and the early exiting idea has been proposed to make trade-offs between model quality and efficiency. This paper aims to address two weaknesses of previous work: (1) existing fine-tuning strategies for early exiting models fail to take full advantage of BERT; (2) meth- ods to make exiting decisions are limited to classification tasks. We propose a more ad- vanced fine-tuning strategy and a learning-to- exit module that extends early exiting to tasks other than classification. Experiments demon- strate improved early exiting for BERT, with better trade-offs obtained by the proposed fine- tuning strategy, successful application to re- gression tasks, and the possibility to combine it with other acceleration methods. Source code can be found at https://github.com/ castorini/berxit. 1 Introduction Large-scale pre-trained language models such as BERT (Devlin et al., 2019) have brought the nat- ural language processing (NLP) community large performance gain but at the cost of heavy com- putational burden. While pre-trained models are available online and fine-tuning is typically done without a strict time budget, inference poses a much lower latency tolerance, and the slow inference speed of these models can impede easy deployment. It becomes even more difficult when inference has to be done on edge devices due to limited network capabilities or privacy concerns. Early exiting (Schwartz et al., 2020; Xin et al., 2020a; Liu et al., 2020) has been proposed to ac- celerate the inference of BERT and models with similar architecture, i.e., those comprising multi- ple transformer layers (Vaswani et al., 2017) with a classifier at the top. Instead of using only one ⋮ ⋮ Transformer 1 Classifier 1 Confidence / LTE Exit Continue Transformer 2 Classifier 2 Confidence / LTE Exit Continue Transformer 𝑛 Classifier 𝑛 Exit Input Figure 1: Multi-output structure of early exiting BERT. classifier, additional classifiers are attached to each transformer layer (see Figure 1), and the entire model is fine-tuned together. At inference time, the sample can perform early exiting through one of the intermediate classifiers. While existing early exiting papers provide promising quality–efficiency trade-offs, improve- ments are necessary for two important components: fine-tuning strategies and exiting decision making. In these papers, fine-tuning strategies are relatively simple and fail to take full advantage of the pre- trained model’s effectiveness; we propose a novel fine-tuning strategy, Alternating, for this multi- output model. Moreover, previous work makes exiting decisions based on the confidence of output probability distributions, and is hence only appli- cable to classification tasks; we extend it to other tasks by proposing the learning-to-exit idea. With carefully designed fine-tuning strategies …

Content truncated to 500 words