Cross-modal Language Generation using Pivot Stabilization for Web-scale Language Coverage

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 160–170 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 160 Cross-modal Language Generation using Pivot Stabilization for Web-scale Language Coverage Ashish V. Thapliyal Google Research asht@google.com Radu Soricut Google Research rsoricut@google.com Abstract Cross-modal language generation tasks such as image captioning are directly hurt in their ability to support non-English languages by the trend of data-hungry models combined with the lack of non-English annotations. We investigate potential solutions for combining existing language-generation annotations in English with translation capabilities in order to create solutions at web-scale in both domain and language coverage. We describe an ap- proach called Pivot-Language Generation Sta- bilization (PLuGS), which leverages directly at training time both existing English anno- tations (gold data) as well as their machine- translated versions (silver data); at run-time, it generates first an English caption and then a corresponding target-language caption. We show that PLuGS models outperform other candidate solutions in evaluations performed over 5 different target languages, under a large- domain testset using images from the Open Im- ages dataset. Furthermore, we find an inter- esting effect where the English captions gen- erated by the PLuGS models are better than the captions generated by the original, mono- lingual English model. 1 Introduction Data hungry state-of-the-art neural models for language generation have the undesired potential to widen the quality gap between English and non-English languages, given the scarcity of non- English labeled data. One notable exception is machine translation, which benefits from large amounts of bilingually or multilingually annotated data. But cross-modal language generation tasks, such as automatic image captioning, tend to be directly hurt by this trend: existing datasets such as Flickr (Young et al., 2014a), MSCOCO (Lin et al., 2014), and Conceptual Captions (Sharma et al., 2018) have extensive labeled data for En- glish, but labeled data is extremely scarce in other languages (Elliott et al., 2016) (at 2 orders of mag- nitude less for a couple of languages, and none for the rest). In this paper, we conduct a study aimed at an- swering the following question: given a large an- notated web-scale dataset such as Conceptual Cap- tions (Sharma et al., 2018) in one language, and a baseline machine translation system, what is the optimal way to scale a cross-modality language generation system to new languages at web-scale? We focus our study on the task of automatic im- age captioning, as a representative for cross-modal language generation where back-and-forth consis- tency cannot be leveraged in a straightforward man- ner 1. In this framework, we proceed to test sev- eral possible solutions, as follows: (a) leverage existing English (En) image captioning datasets to train a model that generates En captions, which are then translated into a target language X; we call this approach Train-Generate-Translate (TGT); (b) leverage existing En captioning datasets and translation capabilities to first translate the data into the target language X, and then train a model that generates X -language captions; we call …

Content truncated to 500 words