Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysis

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 70–79 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 70 Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation Bowen Wu1,2, Huan Zhang1∗, Mengyuan Li1∗, Zongsheng Wang2, Qihang Feng2, Junhong Huang2, Baoxun Wang2 1Peking University, Beijing, China 2Platform and Content Group, Tencent jason wbw,zhanghuan123,limengyuan@pku.edu.cn jasoawang,careyfeng,vincenthuang,asulewang@tencent.com Abstract Recently, BERT has become an essential in- gredient of various NLP deep models due to its effectiveness and universal-usability. How- ever, the online deployment of BERT is of- ten blocked by its large-scale parameters and high computational cost. There are plenty of studies showing that the knowledge distilla- tion is efficient in transferring the knowledge from BERT into the model with a smaller size of parameters. Nevertheless, current BERT distillation approaches mainly focus on task- specified distillation, such methodologies lead to the loss of the general semantic knowl- edge of BERT for universal-usability. In this paper, we propose a sentence representation approximating oriented distillation framework that can distill the pre-trained BERT into a simple LSTM based model without specify- ing tasks. Consistent with BERT, our dis- tilled model is able to perform transfer learn- ing via fine-tuning to adapt to any sentence- level downstream task. Besides, our model can further cooperate with task-specific distillation procedures. The experimental results on mul- tiple NLP tasks from the GLUE benchmark show that our approach outperforms other task- specific distillation methods or even much larger models, i.e., ELMO, with efficiency well-improved. 1 Introduction As one of the most important progress in the Natu- ral Language Processing field recently, the Bidirec- tional Encoder Representation from Transformers (BERT) (Devlin et al., 2019) has been proved to be effective in improving the performances of vari- ous NLP tasks by providing a powerful pre-trained language model based on large-scale unlabeled cor- pora. Recent studies have shown that BERT’s capa- bility can be further enhanced by utilizing deeper ar- chitectures or performing the pre-training on larger ∗* Equal contribution during the internship at Tencent. corpora with appropriate guidance (Radford et al., 2019; Yang et al., 2019; Liu et al., 2019b). Despite its strength in building distributed se- mantic representations of sentences and supporting various NLP tasks, BERT holds a huge amount of parameters that raises the difficulty of conducting online deployment due to its unsatisfying compu- tational efficiency. To address this issue, various studies have been done to utilize the knowledge distillation (Hinton et al., 2015) for compressing BERT and meanwhile keep its semantic model- ing capability as much as possible (Chia et al., 2019; Tsai et al., 2019). The distilling methodolo- gies include simulating BERT with a much smaller model (e.g., LSTM) (Tang et al., 2019b) and re- ducing some of the components, such as transform- ers, attentions to obtain the smaller BERT based model (Sun et al., 2019; Barkan et al., 2019). Nevertheless, the current methods highly rely on a labeled dataset upon …

Content truncated to 500 words