Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5721–5727, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5721 Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing Yuxuan Wang1, Wanxiang Che1∗, Jiang Guo2, Yijia Liu1, and Ting Liu1 1Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology 2Computer Science and Artificial Intelligence Laboratory, MIT {yxwang,car,yjliu,tliu}@ir.hit.edu.cn,jiang guo@csail.mit.edu Abstract This paper investigates the problem of learn- ing cross-lingual representations in a contex- tual space. We propose Cross-Lingual BERT Transformation (CLBT), a simple and effi- cient approach to generate cross-lingual con- textualized word embeddings based on pub- licly available pre-trained BERT models (De- vlin et al., 2018). In this approach, a lin- ear transformation is learned from contextual word alignments to align the contextualized embeddings independently trained in different languages. We demonstrate the effectiveness of this approach on zero-shot cross-lingual transfer parsing. Experiments show that our embeddings substantially outperform the pre- vious state-of-the-art that uses static embed- dings. We further compare our approach with XLM (Lample and Conneau, 2019), a recently proposed cross-lingual language model trained with massive parallel data, and achieve highly competitive results. 1 1 Introduction One of the most promising directions for cross- lingual dependency parsing, which also remains a challenge, is to bridge the gap of lexical features. Prior works (Xiao and Guo, 2014; Guo et al., 2015) have shown that cross-lingual word embed- dings are able to significantly improve the trans- fer performance compared to delexicalized mod- els (McDonald et al., 2011, 2013). These cross- lingual word embeddings are static in the sense that they do not change with the context.2 Recently, contextualized word embeddings de- rived from large-scale pre-trained language mod- els (McCann et al., 2017; Peters et al., 2017, 2018; ∗Email corresponding 1Our code is released at https://github.com/ WangYuxuan93/CLBT 2In this paper, we refer to these embeddings as static as opposed to contextualized ones. WX Y He loves the movie channel Cambiar al Canal 4 (Change to Channel 4) X Y The ship went aground in the channel El canal está marcado por boyas (The channel is marked by buoys) Figure 1: A toy illustration of the method, where con- textualized embeddings of the word canal from Span- ish is transformed to the semantic space of English. Devlin et al., 2018) have demonstrated dramatic superiority over traditional static word embed- dings, establishing new state-of-the-arts in various monolingual NLP tasks (Ili´c et al., 2018; Schus- ter et al., 2018). The success has also been rec- ognized in dependency parsing (Che et al., 2018). The great potential of these contextualized embed- dings has inspired us to extend its power to cross- lingual scenarios. Several recent works have been proposed to learn contextualized cross-lingual embeddings by training cross-lingual language models from scratch with parallel data as supervision, and has been demonstrated effective in several down- stream tasks (Schuster et al., 2018; Mulcaire et al., 2019; Lample and Conneau, 2019). These …

Content truncated to 500 words