Data Annealing for Informal Language Understanding Tasks

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3153–3159 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3153 Data Annealing for Informal Language Understanding Tasks Jing Gu University of California, Davis jkgu@ucdavis.edu Zhou Yu University of California, Davis joyu@ucdavis.edu Abstract There is a huge performance gap between formal and informal language understanding tasks. The recent pre-trained models that im- proved formal language understanding tasks did not achieve a comparable result on infor- mal language. We propose data annealing transfer learning procedure to bridge the per- formance gap on informal natural language un- derstanding tasks. It successfully utilizes a pre- trained model such as BERT in informal lan- guage. In the data annealing procedure, the training set contains mainly formal text data at first; then, the proportion of the informal text data is gradually increased during the train- ing process. Our data annealing procedure is model-independent and can be applied to var- ious tasks. We validate its effectiveness in exhaustive experiments. When BERT is im- plemented with our learning procedure, it out- performs all the state-of-the-art models on the three common informal language tasks. 1 Introduction and Related Work Because of the noisy nature of the informal lan- guage and the shortage of labeled data, the progress on informal language is not as promising as in formal language. Many tasks on formal data ob- tain a high performance due to deep neural models (Peters et al., 2018; Devlin et al., 2018). How- ever, these state-of-the-art models’ excellent per- formance usually fails to transfer to informal data directly. For example, when a BERT model is fine-tuned on informal data, its performance is less encouraging than on formal data. It is because of the domain discrepancy between the pre-training corpus used by BERT and the target data. To solve the issues mentioned above, we propose a model-agnostic data annealing procedure. We set informal data as target data and set formal data as source data. The training data first contains mainly source data, when data annealing procedure takes the advantages of a proper parameter initialization from the clean nature of formal data. The propor- tion of source data keeps decreasing exponentially while the proportion of target data keeps increasing, which empowers the model with more freedom to explore the direction of its next update. The philosophy behind data annealing is shared with other commonly used annealing techniques. One popular usage of annealing is learning rate annealing. A gradually decayed learning rate en- hances the model with more freedom of exploration at the beginning and leads to better model perfor- mance (Zeiler, 2012; Yang and Zhang, 2018; De- vlin et al., 2018). Another widespread implementa- tion of annealing is simulated annealing (Bertsimas and Tsitsiklis, 1993). It reduces the probability of a model converging to a bad local optimal by introducing random noise in the training process. Data annealing has similar functionality with sim- ulated annealing but replaces random noise with source data. By doing this, the model explores more space at …

Content truncated to 500 words