Cross-Lingual Image Caption Generation

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1780–1790, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Cross-Lingual Image Caption Generation Takashi Miyazaki∗ Yahoo Japan Corporation Tokyo, Japan takmiyaz@yahoo-corp.jp Nobuyuki Shimizu∗ Yahoo Japan Corporation Tokyo, Japan nobushim@yahoo-corp.jp Abstract Automatically generating a natural lan- guage description of an image is a fun- damental problem in artificial intelligence. This task involves both computer vision and natural language processing and is called “image caption generation.” Re- search on image caption generation has typically focused on taking in an image and generating a caption in English as ex- isting image caption corpora are mostly in English. The lack of corpora in languages other than English is an issue, especially for morphologically rich languages such as Japanese. There is thus a need for cor- pora sufficiently large for image caption- ing in other languages. We have developed a Japanese version of the MS COCO cap- tion dataset and a generative model based on a deep recurrent architecture that takes in an image and uses this Japanese ver- sion of the dataset to generate a caption in Japanese. As the Japanese portion of the corpus is small, our model was de- signed to transfer the knowledge represen- tation obtained from the English portion into the Japanese portion. Experiments showed that the resulting bilingual compa- rable corpus has better performance than a monolingual corpus, indicating that image understanding using a resource-rich lan- guage benefits a resource-poor language. 1 Introduction Automatically generating image captions by de- scribing the content of an image using natural lan- guage sentences is a challenging task. It is es- pecially challenging for languages other than En- ∗∗ Both authors contributed equally to this work. glish due to the sparsity of annotated resources in the target language. A promising solution to this problem is to create a comparable corpus. To support the image caption generation task in Japanese, we have annotated images taken from the MS COCO caption dataset (Chen et al., 2015b) with Japanese captions. We call our corpus the “YJ Captions 26k Dataset.” While the size of our dataset is comparatively large with 131,740 captions, it greatly trails the 1,026,459 captions in the MS COCO dataset. We were thus moti- vated to transfer the resources in English (source language) to Japanese and thereby improve im- age caption generation in Japanese (target lan- guage). In natural language processing, a task in- volving transferring information across languages is known as a cross-lingual natural language task, and well known tasks include cross-lingual senti- ment analysis (Chen et al., 2015a), cross-lingual named entity recognition (Zirikly and Hagiwara, 2015), cross-lingual dependency parsing (Guo et al., 2015), and cross-lingual information retrieval (Funaki and Nakayama, 2015). Existing work in the cross-lingual setting is usu- ally formulated as follows. First, to overcome the language barrier, create a connection between the source and target languages, generally by using a dictionary or parallel corpus. Second, develop an appropriate knowledge transfer approach to lever- age the …

Content truncated to 500 words