TF-LM: TensorFlow-based Language Modeling Toolkit

TF-LM: TensorFlow-based Language Modeling Toolkit Lyan Verwimp, Hugo Van hamme, Patrick Wambacq ESAT – PSI, KU Leuven Kasteelpark Arenberg 10, 3001 Heverlee, Belgium {lyan.verwimp, hugo.vanhamme, patrick.wambacq}@esat.kuleuven.be Abstract Recently, an abundance of deep learning toolkits has been made freely available. These toolkits typically offer the building blocks and sometimes simple example scripts, but designing and training a model still takes a considerable amount of time and knowledge. We present language modeling scripts based on TensorFlow that allow one to train and test competitive models directly, by using a pre-defined configuration or changing it to their needs. There are several options for input features (words, characters, words combined with characters, character n-grams) and for batching (sentence- or discourse-level). The models can be used to test the perplexity, predict the next word(s), re-score hypotheses or generate debugging files for interpolation with n-gram models. Addi- tionally, we make available LSTM language models trained on a variety of Dutch texts and English benchmarks, that can be used immediately, thereby avoiding the time and computationally expensive training process. The toolkit is open source and can be found at https://github.com/lverwimp/tf-lm. Keywords: language modeling, LSTM, deep learning, toolkit 1. Introduction Language models (LMs) play a crucial role in many speech and language processing tasks, among others speech recog- nition, machine translation and optical character recogni- tion. The current state of the art are recurrent neural net- work (RNN) based LMs (Mikolov et al., 2010), and more specifically long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) LMs (Sundermeyer et al., 2012). Building, training, optimizing and testing these networks from scratch would require a huge amount of expertise and time. There exist many deep learning frameworks that offer the building blocks: TensorFlow (Abadi et al., 2015), Keras (Chollet, 2015), Torch (Collobert et al., 2011), Theano (Theano Development Team, 2016), Caffe (Jia et al., 2014) and others. Researchers proposing a new type of model also frequently publish their code, but typically do not offer a more general framework. Among the deep learning frameworks, TensorFlow is ar- guably the most widely used, as the Github statistics in ta- ble 1. demonstrate. However, as far as we know (see sec- tion 2. for a more detailed description of the existing tools), there does not exist a toolkit that allows one to quickly de- sign, train and test their own ‘baseline’ LMs with Tensor- Flow. We release a toolkit that contains modular code that should be easy to adapt, standard recipes to train compet- itive baseline models that can be adapted in a simple con- figuration file and LMs that are pre-trained on a variety of English benchmarks and corpora of spoken Dutch. Framework Stars Forks TensorFlow 86,940 33,924 Caffe 20,003 12,275 Keras 19,261 6,978 Torch 7,232 2,140 Theano 6,862 2,277 Table 1: Github statistics for several deep learning frame- works, recorded on Sept 6, 2017. In the remainder of this paper, we discuss other toolkits that provide language modeling scripts in section 2., de- scribe the documentation and functionality that our toolkit provides …

Content truncated to 500 words