iParaphrasing: Extracting Visually Grounded Paraphrases via an Image

Proceedings of the 27th International Conference on Computational Linguistics, pages 3479–3492 Santa Fe, New Mexico, USA, August 20-26, 2018. 3479 iParaphrasing: Extracting Visually Grounded Paraphrases via an Image Chenhui Chu,1 Mayu Otani2 and Yuta Nakashima1 1Institute for Datability Science, Osaka University 2CyberAgent, Inc. chu,n-yuta@ids.osaka-u.ac.jp, otani mayu@cyberagent.co.jp Abstract A paraphrase is a restatement of the meaning of a text in other words. Paraphrases have been studied to enhance the performance of many natural language processing tasks. In this paper, we propose a novel task iParaphrasing to extract visually grounded paraphrases (VGPs), which are different phrasal expressions describing the same visual concept in an image. These extracted VGPs have the potential to improve language and image multimodal tasks such as visual question answering and image captioning. How to model the similarity between VGPs is the key of iPara- phrasing. We apply various existing methods as well as propose a novel neural network-based method with image attention, and report the results of the first attempt toward iParaphrasing. 1 Introduction A paraphrase is a restatement of the meaning of a word, phrase, or sentence within the context of a specific language (e.g., “a red jersey” and “a red uniform shirt” in Figure 1 are paraphrases) (Bhagat and Hovy, 2013). Paraphrases have been exploited for natural language understanding, and shown to be very effective for various natural language processing (NLP) tasks, including question answering (Riezler et al., 2007), summarization (Zhou et al., 2006), machine translation (Chu and Kurohashi, 2016), text normalization (Ling et al., 2013), textual entailment recognition (Androutsopoulos and Malakasiotis, 2010), and semantic parsing (Berant and Liang, 2014). In this paper, we propose a novel task named, iParaphrasing, to extract visually grounded paraphrases (VGPs). We define VGPs as different phrasal expressions that describe the same visual concept in an image. Nowadays, with the spread of the web and social media, it is easy to collect large amounts of images with their describing text. For example, different news sites release news with the same topic using the same image; photos with many comments are posted to social networking sites and blogs. As these describing texts are written by different people but about the same image, there are potentially large amounts of VGPs in the describing text (Figure 1). We aim to accurately extract these paraphrases using the image as a pivot to associate different phrases. The extracted VGPs can be applied to various computer vision (CV) and NLP tasks, such as image captioning (Vinyals et al., 2015) and visual question answering (VQA) (Wu et al., 2017), for the better understanding of both images and languages. For example, a VQA system must understand queries of different expressions about the same visual concept (e.g., “a male” and “the pitcher” in Figure 1) in order to answer a question properly. VGPs can also be applied to the evaluation of image captioning systems in the similar way as paraphrases have been applied for machine translation evaluation (Snover et al., 2009). As a pioneering study, we work on iParaphrasing on …

Content truncated to 500 words