Multilingual Constituency Parsing with Self-Attention and Pre-Training

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499–3505 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3499 Multilingual Constituency Parsing with Self-Attention and Pre-Training Nikita Kitaev Steven Cao Computer Science DivisionDan Klein University of California, Berkeley {kitaev,stevencao,klein}@berkeley.edu Abstract We show that constituency parsing benefits from unsupervised pre-training across a vari- ety of languages and a range of pre-training conditions. We first compare the benefits of no pre-training, fastText (Bojanowski et al., 2017; Mikolov et al., 2018), ELMo (Peters et al., 2018), and BERT (Devlin et al., 2018a) for English and find that BERT outperforms ELMo, in large part due to increased model capacity, whereas ELMo in turn outperforms the non-contextual fastText embeddings. We also find that pre-training is beneficial across all 11 languages tested; however, large model sizes (more than 100 million parameters) make it computationally expensive to train separate models for each language. To address this shortcoming, we show that joint multilingual pre-training and fine-tuning allows sharing all but a small number of parameters between ten languages in the final model. The 10x reduc- tion in model size compared to fine-tuning one model per language causes only a 3.2% rel- ative error increase in aggregate. We further explore the idea of joint fine-tuning and show that it gives low-resource languages a way to benefit from the larger datasets of other lan- guages. Finally, we demonstrate new state-of- the-art results for 11 languages, including En- glish (95.8 F1) and Chinese (91.8 F1). 1 Introduction There has recently been rapid progress in devel- oping contextual word representations that im- prove accuracy across a range of natural language tasks (Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018a). While we have shown in previous work (Kitaev and Klein, 2018) that such representations are ben- eficial for constituency parsing, our earlier results only consider the LSTM-based ELMo representa- tions (Peters et al., 2018), and only for the English language. In this work, we study a broader range of pre-training conditions and experiment over a variety of languages, both jointly and individually. First, we consider the impact on parsing of us- ing different methods for pre-training initial net- work layers on a large collection of un-annotated text. Here, we see that pre-training provides benefits for all languages evaluated, and that BERT (Devlin et al., 2018a) outperforms ELMo, which in turn outperforms fastText (Bojanowski et al., 2017; Mikolov et al., 2018), which performs slightly better than the non pre-trained baselines. Pre-training with a larger model capacity typically leads to higher parsing accuracies. Second, we consider various schemes for the parser fine-tuning that is required after pre- training. While BERT itself can be pre-trained jointly on many languages, successfully applying it, e.g. to parsing, requires task-specific adaptation via fine-tuning (Devlin et al., 2018a). Therefore, the obvious approach to parsing ten languages is to fine-tune ten times, producing ten variants of the parameter-heavy BERT layers. In this work, we …

Content truncated to 500 words