Low-Resource Sequence Labeling via Unsupervised Multilingual Contextualized Representations

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1028–1039, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1028 Low-Resource Sequence Labeling via Unsupervised Multilingual Contextualized Representations Zuyi Bao‡, Rui Huang♦, Chen Li‡∗and Kenny Q. Zhu† ‡Alibaba Group ♦Zhejiang University, China †Shanghai Jiao Tong University, Shanghai, China ‡{zuyi.bzy,puji.lc}@alibaba-inc.com ♦iurgnauh@zju.edu.cn, †kzhu@cs.sjtu.edu.cn Abstract Previous work on cross-lingual sequence la- beling tasks either requires parallel data or bridges the two languages through word-by- word matching. Such requirements and as- sumptions are infeasible for most languages, especially for languages with large linguis- tic distances, e.g., English and Chinese. In this work, we propose a Multilingual Lan- guage Model with deep semantic Alignment (MLMA) to generate language-independent representations for cross-lingual sequence la- beling. Our methods require only monolingual corpora with no bilingual resources at all and take advantage of deep contextualized repre- sentations. Experimental results show that our approach achieves new state-of-the-art NER and POS performance across European lan- guages, and is also effective on distant lan- guage pairs such as English and Chinese.1 1 Introduction Sequence labeling tasks such as named entity recognition (NER) and part-of-speech (POS) tag- ging are fundamental problems in natural lan- guage processing (NLP). Recent sequence label- ing models achieve state-of-the-art performance by combining both character-level and word-level information (Chiu and Nichols, 2016; Ma and Hovy, 2016; Lample et al., 2016). However, these models heavily rely on large-scale annotated train- ing data, which may not be available in most lan- guages. Cross-lingual transfer learning is pro- posed to address the label scarcity problem by transferring annotations from high-resource lan- guages (source languages) to low-resource lan- guages (target languages). In this scenario, a ma- ∗Corresponding Author. † Kenny Q. Zhu was partially supported by NSFC grant 91646205 and Alibaba visiting scholar program. 1The code is released at https://github.com/ baozuyi/MLMA. jor challenge is how to bridge interlingual gaps with modest resource requirements. There is a large body of work exploring cross- lingual transfer through language-independent features, such as morphological features and uni- versal POS tags for cross-lingual NER (Tsai et al., 2016) and dependency parsers (McDonald et al., 2011). However, these approaches require linguis- tic knowledge for language-independent feature engineering, which is expensive in low-resource settings. Other work relies on bilingual resources to transfer knowledge from source languages to target languages. Parallel corpora are widely used to project annotations from the source to the tar- get side (Yarowsky et al., 2001; Ehrmann et al., 2011; Kim et al., 2012; Wang and Manning, 2014). These methods could achieve strong performance with a large amount of bilingual data, which is scarce in low-resource settings. Recent research leverages cross-lingual word embeddings (CLWEs) to establish inter-lingual connections and reduce the requirements of par- allel data to a small lexicon or even no bilin- gual resource (Ni et al., 2017; Fang and Cohn, 2017; Xie et al., 2018). However, word embedding spaces may not be completely isomorphic due to language-specific …

Content truncated to 500 words