Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1355–1360, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1355 Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER Phillip Keung, Yichao Lu, Vikas Bhardwaj Amazon Inc. {keung, yichaolu, vikab}@amazon.com Abstract Contextual word embeddings (e.g. GPT, BERT, ELMo, etc.) have demonstrated state- of-the-art performance on various NLP tasks. Recent work with the multilingual version of BERT has shown that the model performs very well in cross-lingual settings, even when only labeled English data is used to finetune the model. We improve upon multilingual BERT’s zero-resource cross-lingual performance via adversarial learning. We report the magnitude of the improvement on the multilingual ML- Doc text classification and CoNLL 2002/2003 named entity recognition tasks. Furthermore, we show that language-adversarial training en- courages BERT to align the embeddings of En- glish documents and their translations, which may be the cause of the observed performance gains. 1 Introduction Contextual word embeddings (Devlin et al., 2019; Peters et al., 2018; Radford et al., 2019) have been successfully applied to various NLP tasks, includ- ing named entity recognition, document classifi- cation, and textual entailment. The multilingual version of BERT (which is trained on Wikipedia articles from 100 languages and equipped with a 110,000 shared wordpiece vocabulary) has also demonstrated the ability to perform ‘zero- resource’ cross-lingual classification on the XNLI dataset (Conneau et al., 2018). Specifically, when multilingual BERT is finetuned for XNLI with En- glish data alone, the model also gains the ability to handle the same task in other languages. We be- lieve that this zero-resource transfer learning can be extended to other multilingual datasets. In this work, we explore BERT’s1 zero-resource performance on the multilingual MLDoc classi- fication and CoNLL 2002/2003 NER tasks. We 1‘BERT’ hereafter refers to multilingual BERT find that the baseline zero-resource performance of BERT exceeds the results reported in other work, even though cross-lingual resources (e.g. parallel text, dictionaries, etc.) are not used dur- ing BERT pretraining or finetuning. We apply adversarial learning to further improve upon this baseline, achieving state-of-the-art zero-resource results. There are many recent approaches to zero- resource cross-lingual classification and NER, in- cluding adversarial learning (Chen et al., 2019; Kim et al., 2017; Xie et al., 2018; Joty et al., 2017), using a model pretrained on parallel text (Artetxe and Schwenk, 2018; Lu et al., 2018; Lample and Conneau, 2019) and self-training (Hajmohammadi et al., 2015). Due to the newness of the sub- ject matter, the definition of ‘zero-resource’ varies somewhat from author to author. For the experi- ments that follow, ‘zero-resource’ means that, dur- ing model training, we do not use labels from non- English data, nor do we use human or machine- generated parallel text. Only labeled English text and unlabeled non-English text are used during training, and hyperparameters are selected using English evaluation sets. Our contributions are the following: • We demonstrate that the addition …

Content truncated to 500 words