A Call for More Rigor in Unsupervised Cross-lingual Learning

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7375–7388 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7375 A Call for More Rigor in Unsupervised Cross-lingual Learning Mikel Artetxe†∗, Sebastian Ruder‡∗, Dani Yogatama‡, Gorka Labaka†, Eneko Agirre† †HiTZ Center, University of the Basque Country (UPV/EHU) ‡DeepMind {mikel.artetxe,gorka.labaka,e.agirre}@ehu.eus {ruder,dyogatama}@google.com Abstract We review motivations, definition, approaches, and methodology for unsupervised cross- lingual learning and call for a more rigorous position in each of them. An existing rationale for such research is based on the lack of par- allel data for many of the world’s languages. However, we argue that a scenario without any parallel data and abundant monolingual data is unrealistic in practice. We also discuss differ- ent training signals that have been used in pre- vious work, which depart from the pure unsu- pervised setting. We then describe common methodological issues in tuning and evalua- tion of unsupervised cross-lingual models and present best practices. Finally, we provide a unified outlook for different types of research in this area (i.e., cross-lingual word embed- dings, deep multilingual pretraining, and un- supervised machine translation) and argue for comparable evaluation of these models. 1 Introduction The study of the connection among human lan- guages has contributed to major discoveries in- cluding the evolution of languages, the reconstruc- tion of proto-languages, and an understanding of language universals (Eco and Fentress, 1995). In natural language processing, the main promise of multilingual learning is to bridge the digital lan- guage divide, to enable access to information and technology for the world’s 6,900 languages (Ruder et al., 2019). For the purpose of this paper, we define “multilingual learning” as learning a com- mon model for two or more languages from raw text, without any downstream task labels. Common use cases include translation as well as pretraining multilingual representations. We will use the term interchangeably with “cross-lingual learning”. ∗Equal contribution. Recent work in this direction has increasingly focused on purely unsupervised cross-lingual learn- ing (UCL)—i.e., cross-lingual learning without any parallel signal across the languages. We provide an overview in §2. Such work has been motivated by the apparent dearth of parallel data for most of the world’s languages. In particular, previous work has noted that “data encoding cross-lingual equivalence is often expensive to obtain” (Zhang et al., 2017a) whereas “monolingual data is much easier to find” (Lample et al., 2018a). Overall, it has been argued that unsupervised cross-lingual learning “opens up opportunities for the processing of extremely low-resource languages and domains that lack parallel data completely” (Zhang et al., 2017a). We challenge this narrative and argue that the scenario of no parallel data and sufficient monolin- gual data is unrealistic and not reflected in the real world (§3.1). Nevertheless, UCL is an important research direction and we advocate for its study based on an inherent scientific interest (to better understand and make progress on general language understanding), usefulness as a lab setting, and simplicity (§3.2). Unsupervised cross-lingual learning permits no supervisory signal by …

Content truncated to 500 words