Adding Syntactic Annotations to Flickr30k Entities Corpus for Multimodal Ambiguous Prepositional-Phrase Attachment Resolution

Adding Syntactic Annotations to Flickr30k Entities Corpus for Multimodal Ambiguous Prepositional-Phrase Attachment Resolution Sebastien Delecraz, Alexis Nasr, Frederic Bechet, Benoit Favre Aix-Marseille Univ, Universit´e de Toulon, CNRS, LIS 163 avenue de Luminy - Case 901 F-13288 Marseille - Cedex 9 - FRANCE {firstname.lastname}@univ-amu.fr Abstract We propose in this paper to add to the captions of the Flickr30k Entities corpus some syntactic annotations in order to study the joint processing of image and language features for the Preposition-Phrase attachment disambiguation task. The annotation has been performed on the English version of the captions and automatically projected on their French and German translations. Keywords: PP-attachments, Multimodal Corpus, Multilingual Corpus 1. Introduction Joint processing of image and text is a very active area of research. It is studied mostly in the context of natural lan- guage generation, for example for generating a textual de- scription of a video or an image. Recently open-domain language generation from images or videos received a lot of attention through the use of multimodal deep neural net- works (Vinyals et al., 2015). Theses models build a unified representation for both image and language features and generate in an end-to-end process a text directly from an image, without an explicit representation (syntactic or se- mantic) of the text generated. In this paper we propose to study joint image and language processing for language parsing rather than generation. The main idea of this study is to check if the use of visual fea- tures extracted from an image can be useful in order to dis- ambiguate the linguistic analysis of a caption that describes the same image. To do so we propose a new framework for testing multimodal approaches on the specific task of am- biguous Prepositional-Phrase attachment (PP-attachment) resolution. PP-attachments are known to be an important source of er- rors in parsing natural language. The main reason being that, in many cases, correct attachments cannot be predicted accurately based on pure syntactic considerations: their prediction ask for precise lexical co-occurrences. Such in- formation is usually not found in treebanks that are lim- ited in their size and therefore do not model many bi-lexical phenomena. Besides, disambiguation may ask for non lin- guistic knowledge which is not present in treebanks. In this paper, we propose to create a corpus for support- ing PP-attachment disambiguation research by combining textual and visual information. The contribution of this study is the selection and the manual annotation of a corpus of ambiguous PP-attachments from the multimodal corpus Flickr30k Entities (Plummer et al., 2017). A full parse of the sentences containing a hand-corrected PP-attachment, which is compatible with the manual attachment is also produced. In addition, we use MT alignments to transfer the annotations to French and German translations from the Multi30k corpus (Elliott et al., 2016; Elliott et al., 2017). Figure 1: Example of the F30kE annotations. The image is described with five captions, each annotated with entities. Entities that corefer with a visual element in the image are linked to the corresponding bounding …

Content truncated to 500 words