Small and Practical BERT Models for Sequence Labeling

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3632–3636, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3632 Small and Practical BERT Models for Sequence Labeling Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li and Amelia Archer Google Research San Francisco, CA, USA {henrytsai,riesa,melvinp,navari,xinl,ameliaarcher}@google.com Abstract We propose a practical scheme to train a sin- gle multilingual sequence labeling model that yields state of the art results and is small and fast enough to run on a single CPU. Starting from a public multilingual BERT checkpoint, our final model is 6x smaller and 27x faster, and has higher accuracy than a state-of-the- art multilingual baseline. We show that our model especially outperforms on low-resource languages, and works on codemixed input text without being explicitly trained on codemixed examples. We showcase the effectiveness of our method by reporting on part-of-speech tag- ging and morphological prediction on 70 tree- banks and 48 languages. 1 Introduction There have been many recent modeling improve- ments (Smith et al., 2018; Bohnet et al., 2018) on morphosyntactic tagging tasks. However, these models have largely focused on building separate models for each language or for a small group of related languages. In this paper, we con- sider the implications of training, evaluating, and deploying a single multilingual model for a di- verse set of almost 50 languages, evaluating on both part-of-speech tagging and morphological at- tribute prediction data from the Universal Depen- dencies repository (Nivre et al., 2018). There are several benefits of using one multilin- gual model over several language specific models. • Parameter sharing among languages reduces model size and enables cross-lingual transfer learning. We show that this improves accu- racy, especially for low-resource languages. • No language identification model needed to decide which language-specific model to query. Critically, this reduces system com- plexity and prevents prediction errors from language identification from propagating into the downstream system. • Multilingual models can be applied to multi- lingual or codemixed inputs without explic- itly being trained on codemixed labeled ex- amples. Otherwise, given, e.g. a mixed Hindi/English input, one must decide to query either the Hindi model or the English model, both of which are sub-optimal. In this paper, we show that by finetuning a pretrained BERT (Devlin et al., 2019) model we can build a multilingual model that has compara- ble or better accuracy to state-of-the-art language- specific models, and outperforms the state-of-the- art on low-resource languages. Our model out- performs other multilingual model baselines by a large margin. We evaluate on both part-of- speech tagging and morphological attribute pre- diction tasks with data from the Universal Depen- dencies repository (Nivre et al., 2018). However, this model is slow, very large, and difficult to de- ploy in practice. We describe our solution for making this model small and practical enough to use in practice on a single CPU, while preserving quality. The final model is 27x faster …

Content truncated to 500 words