German’s Next Language Model

Proceedings of the 28th International Conference on Computational Linguistics, pages 6788–6796 Barcelona, Spain (Online), December 8-13, 2020 6788 German’s Next Language Model Branden Chan∗†, Stefan Schweter∗‡, Timo M¨oller† †deepset {branden.chan, timo.moeller}@deepset.ai ‡Bayerische Staatsbibliothek M¨unchen Digital Library/Munich Digitization Center stefan.schweter@bsb-muenchen.de Abstract In this work we present the experiments which lead to the creation of our BERT and ELECTRA based German language models, GBERT and GELECTRA. By varying the input training data, model size, and the presence of Whole Word Masking (WWM) we were able to attain SoTA performance across a set of document classification and named entity recognition (NER) tasks for both models of base and large size. We adopt an evaluation driven approach in training these models and our results indicate that both adding more data and utilizing WWM improve model performance. By benchmarking against existing German models, we show that these models are the best German models to date. Our trained models will be made publicly available to the research community. 1 Introduction Deep transformer based language models have shown state-of-the-art results for various Natural Lan- guage Processing tasks like text classification, NER and question answering (Devlin et al., 2019). They are pretrained, first by feeding in large unlabeled text corpora before being fine-tuned on the down- stream task. In this work we present a set of German BERT and ELECTRA models, the best of which, GELECTRALarge, significantly improves upon state of the art performance on the GermEval18 hate speech detection task by about +4% / +2.5% for the coarse and fine variants of the task respectively. This model also reaches SoTA on the GermEval14 NER task, outperforming the previous best by over +4%. While performant, such models are prohibitively large for many and so we also present a new GBERT model which matches deepset BERT, the previous best German BERT, in size but outperforms it by +2.23% F1 averaged over three tasks. In the process of pretraining the language models, we also a) quantify the effect of increasing the training data by an order of magnitude and b) verify that whole word masking has a positive effect on BERT models. Because of the computational expense of training large language models from scratch, we adopt a downstream-oriented evaluation approach to ensure that we get the best performance from a limited number of runs. This involves regularly checkpointing the model over the course of pretraining, evalu- ating these on a set of classification and NER tasks and selecting as final the checkpoint which shows the best performance. This stands in contrast to approaches where the final model is simply saved after a fixed number of steps. Our method is also an important tool in diagnosing pretraining and we hope that it will be of use to other teams looking to train effective language models on a budget. 2 Related work Modern language model architectures are trained to build word representations that take into consid- eration the context around a given word. First versions such as ELMo (Peters et al., 2018), …

Content truncated to 500 words