ULMFiT replication

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5579–5587 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5579 Text Classification Using Language Modeling: Reproducing ULMFiT Mohamed Abdellatif, Ahmed Elgammal Rutgers University - Computer Science Piscataway, NJ, USA {mma215, elgammal}@cs.rutgers.edu Abstract In this paper, we reproduce some of the experiments of text classification by fine tuning pre-trained language model on the six English data-sets described in Howard and Ruder (2018) (verification). Then we investigate applicability of the model as is (pre-trained on English) by conducting additional experiments on three other non-English data-sets that are not in the original paper (extension). For the verification experiments, we didn’t generate the exact same numbers as the original paper, however, the replication results are in the same range as compared to the baselines reported for comparison purposes. We attribute this to the limitation in computational resources which forced us to run on smaller batch sizes and for fewer number of epochs. Otherwise, we followed in the footsteps of the author to the best of our abilities (e.g. the libraries1 , tutorials2 , hyper-parameters and transfer learning methodology). We report implementation details as well as lessons learned in the appendices. Keywords: language models, text classification, transfer learning 1. Introduction Transfer learning can be defined as making use of the knowledge gained by solving a source problem Ts towards solving another (target) problem Tt. It is either transduc- tive (the data of the target problem is unlabeled) or induc- tive (the data of the source problem is unlabeled). Defini- tions from Ruder et al. (2019). Inductive transfer learning has been well studied in Computer Vision e.g. Long et al. (2015) and Sharif Razavian et al. (2014). However, not as much in Natural Language Understanding/Processing. The problem of English text classification is motivated by prac- tical applications like anomaly detection, security and le- gal applications. Solving the problem in other languages opens up avenues for these applications in the respective languages. In Computer Vision, works to visualize the filters of a trained Convolutional Neural Network e.g. Mahendran and Vedaldi (2016), Samek et al. (2016) and Yosinski et al. (2014) show that the earlier layers (those closer to the in- put than the output) capture features that general across dif- ferent datasets (e.g. edges, contours ..etc) while later lay- ers (those closer to the output than the input) capture more dataset-specific features. A successful strategy for Com- puter Vision transfer learning has been transferring the ear- lier layers of a model that was trained on a general dataset to a model yet to be trained on a specific dataset Sharif Raza- vian et al. (2014). In Natural Language Processing, there has been a recent breakthrough inspired by transfer learn- ing in Computer Vision. Howard and Ruder (2018) and Radford et al. (2018) are two notable works in this direc- tion. Applying transfer learning by fine-tuning a language model and a classifier significantly outperforms the state of the art results …

Content truncated to 500 words