“Wikily” Supervised Neural Translation Tailored to Cross-Lingual Tasks

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1655–1670 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1655 “Wikily” Supervised Neural Translation Tailored to Cross-Lingual Tasks Mohammad Sadegh Rasooli1∗Chris Callison-Burch2 Derry Tanti Wijaya3 1Microsoft 2Department of Computer and Information Science, University of Pennsylvania 3Department of Computer Science, Boston University mrasooli@microsoft.com, ccb@seas.upenn.edu, wijaya@bu.edu Abstract We present a simple but effective approach for leveraging Wikipedia for neural machine trans- lation as well as cross-lingual tasks of image captioning and dependency parsing without us- ing any direct supervision from external paral- lel data or supervised models in the target lan- guage. We show that first sentences and titles of linked Wikipedia pages, as well as cross- lingual image captions, are strong signals for a seed parallel data to extract bilingual dictio- naries and cross-lingual word embeddings for mining parallel text from Wikipedia. Our fi- nal model achieves high BLEU scores that are close to or sometimes higher than strong su- pervised baselines in low-resource languages; e.g. supervised BLEU of 4.0 versus 12.1 from our model in English-to-Kazakh. More- over, we tailor our “wikily” supervised trans- lation models to unsupervised image caption- ing, and cross-lingual dependency parser trans- fer. In image captioning, we train a multi- tasking machine translation and image cap- tioning pipeline for Arabic and English from which the Arabic training data is a translated version of the English captioning data, using our wikily-supervised translation models. Our captioning results on Arabic are slightly better than that of its supervised model. In depen- dency parsing, we translate a large amount of monolingual text, and use it as artificial train- ing data in an annotation projection frame- work. We show that our model outperforms recent work on cross-lingual transfer of depen- dency parsers. 1 Introduction Developing machine translation models without us- ing bilingual parallel text is an intriguing research problem with real applications: obtaining a large volume of parallel text for many languages is hard if not impossible. Moreover, translation models ∗Research was conducted at The University of Pennsyl- vania. could be used in downstream cross-lingual tasks in which annotated data does not exist for some languages. There has recently been a great deal of interest in unsupervised neural machine trans- lation (e.g. Artetxe et al. (2018a); Lample et al. (2018a,c); Conneau and Lample (2019); Song et al. (2019a); Kim et al. (2020); Tae et al. (2020)). Un- supervised neural machine translation models of- ten perform nearly as well as supervised models when translating between similar languages, but they fail to perform well in low-resource or dis- tant languages (Kim et al., 2020) or out-of-domain monolingual data (Marchisio et al., 2020). In prac- tice, the highest need for unsupervised models is to expand beyond high resource, similar European language pairs. There are two key goals in this paper: Our first goal is developing accurate translation models for low-resource distant languages without any supervi- sion from a supervised model or gold-standard par- allel data. Our second goal is to show …

Content truncated to 500 words