Transfer Fine-Tuning: A BERT Case Study

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5393–5404, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5393 Transfer Fine-Tuning: A BERT Case Study Yuki Arase1⋆ and Junichi Tsujii⋆2 1Osaka University, Japan ⋆Artificial Intelligence Research Center (AIRC), AIST, Japan 2NaCTeM, School of Computer Science, University of Manchester, UK arase@ist.osaka-u.ac.jp, j-tsujii@aist.go.jp Abstract A semantic equivalence assessment is defined as a task that assesses semantic equivalence in a sentence pair by binary judgment (i.e., para- phrase identification) or grading (i.e., seman- tic textual similarity measurement). It consti- tutes a set of tasks crucial for research on nat- ural language understanding. Recently, BERT realized a breakthrough in sentence represen- tation learning (Devlin et al., 2019), which is broadly transferable to various NLP tasks. While BERT’s performance improves by in- creasing its model size, the required compu- tational power is an obstacle preventing prac- tical applications from adopting the technol- ogy. Herein, we propose to inject phrasal paraphrase relations into BERT in order to generate suitable representations for seman- tic equivalence assessment instead of increas- ing the model size. Experiments on standard natural language understanding tasks confirm that our method effectively improves a smaller BERT model while maintaining the model size. The generated model exhibits supe- rior performance compared to a larger BERT model on semantic equivalence assessment tasks. Furthermore, it achieves larger per- formance gains on tasks with limited training datasets for fine-tuning, which is a property desirable for transfer learning. 1 Introduction Paraphrase identification and semantic textual similarity (STS) measurements aim to assess se- mantic equivalence in sentence pairs. These tasks are central problems in natural language under- standing research and its applications. In this pa- per, these tasks are defined as semantic equiva- lence assessments. Sentence representation learning is the basis of assessing semantic equivalence. Unsupervised learning is becoming the preferred approach be- cause it only requires plain corpora, which are now abundantly available. In this approach, a model is pre-trained to generate generic sentence repre- sentations that are broadly transferable to various natural language processing (NLP) tasks. Subse- quently, it is fine-tuned to generate specific repre- sentations for solving a target task using an anno- tated corpus. Considering the high costs of anno- tation, a pre-trained model that efficiently fits the target task with a smaller amount of annotated cor- pus is desired. Recently, Bidirectional Encoder Representa- tions from Transformers (BERT) realized a break- through, which dramatically improved sentence representation learning (Devlin et al., 2019). BERT pre-trains its encoder using language mod- eling and by discriminating surrounding sentences in a document from random ones. Pre-training in this manner allows distributional relations be- tween sentences to be learned. Intensive efforts are currently being made to pre-train larger models by feeding them enormous corpora for improve- ment (Radford et al., 2019; Yang et al., 2019). For example, a large model of BERT has 340M parameters, which is 3.1 times larger than its smaller alternative. …

Content truncated to 500 words