Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1950–1967 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1950 Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction Mahsa Yarmohammadi1∗, Shijie Wu1∗, Marc Marone1, Haoran Xu1, Seth Ebner1, Guanghui Qin1, Yunmo Chen1, Jialiang Guo1, Craig Harman1, Kenton Murray1, Aaron Steven White2, Mark Dredze1, Benjamin Van Durme1 1Johns Hopkins University, 2University of Rochester {mahsa,shijie.wu,vandurme}@jhu.edu Abstract Zero-shot cross-lingual information extraction (IE) describes the construction of an IE model for some target language, given existing an- notations exclusively in some other language, typically English. While the advance of pre- trained multilingual encoders suggests an easy optimism of "train on English, run on any language", we find through a thorough ex- ploration and extension of techniques that a combination of approaches, both new and old, leads to better performance than any one cross- lingual strategy in particular. We explore techniques including data projection and self- training, and how different pretrained encoders impact them. We use English-to-Arabic IE as our initial example, demonstrating strong per- formance in this setting for event extraction, named entity recognition, part-of-speech tag- ging, and dependency parsing. We then apply data projection and self-training to three tasks across eight target languages. Because no sin- gle set of techniques performs the best across all tasks, we encourage practitioners to explore various configurations of the techniques de- scribed in this work when seeking to improve on zero-shot training. 1 Introduction We consider zero-shot cross-lingual information extraction (IE), in which training data exists in a source language but not in a target language. Massively multilingual encoders like Multilingual BERT (mBERT; Devlin et al., 2019) and XLM- RoBERTa (XLM-R; Conneau et al., 2020a) allow for a strategy of training only on the source lan- guage data, trusting entirely in a shared underly- ing feature representation across languages (Wu and Dredze, 2019; Conneau et al., 2020b). How- ever, in meta-benchmarks like XTREME (Hu et al., 2020), such cross-lingual performance on struc- tured prediction tasks is far behind that on sentence- ∗Equal contribution level or retrieval tasks (Ruder et al., 2021): perfor- mance in the target language is often far below that of the source language. Before multilingual encoders, cross-lingual IE was approached largely as a data projection problem: one either translated and aligned the source training data to the target language, or at test time one translated target lan- guage inputs to the source language for prediction (Yarowsky and Ngai, 2001). We show that by augmenting the source lan- guage training data with data in the target language—either via projection of the source data to the target language (so-called “silver” data) or via self-training with translated text—zero-shot per- formance can be improved. Further improvements might come from using better pretrained encoders or improving on a projection strategy through bet- ter automatic translation models or better align- ment models. In this paper, we explore all the options above, finding that everything is all it takes to achieve our best …

Content truncated to 500 words