The Effects of Unimodal Representation Choices on Multimodal Learning

The Effects of Unimodal Representation Choices on Multimodal Learning Fernando Tadao Ito, Helena de Medeiros Caseli, Jander Moreira Universidade Federal de S˜ao Carlos Rod. Washington Lu´ıs Km 235, Postal Box 676, CEP: 13565-905, S˜ao Carlos-SP f.tadaoito@gmail.com, helenacaseli@dc.ufscar.br, jander@dc.ufscar.br Abstract Multimodal representations are distributed vectors that map multiple modes of information to a single mathematical space, where distances between instances delineate their similarity. In most cases, using a single unimodal representation technique is sufficient for each mode in the creation of multimodal spaces. In this paper, we investigate how different unimodal representations can be combined, and argue that the way they are combined can affect the performance, representation accuracy and classification metrics of other multimodal methods. In the experiments present in this paper, we used a dataset composed of images and text descriptions of products that have been extracted from an e-commerce site in Brazil. From this dataset, we tested our hypothesis in common classification problems to evaluate how multimodal representations can differ according to their component unimodal representation methods. For this domain, we selected eight methods of unimodal representation: LSI, LDA, Word2Vec, GloVe for text; SIFT, SURF, ORB and VGG19 for images. Multimodal representations were built by a multimodal deep autoencoder and a bidirectional deep neural network. Keywords: distributed representations, multimodal representation, multimodal autoencoders, classification 1. Introduction In the real world, multiple modes of information are gath- ered to create knowledge in a way humans can understand. From structured language, we can express abstract ideas in a standardized fashion; while visualization can help us to detect and observe objects and intention. However, in mathematical terms, each kind of data has different intrin- sic statistical features that cannot be compared in a trivial way. To analyze these features, we use distributed representa- tion models to map real-world unimodal data to mathe- matical vectors with reduced dimensionality (Van Gelder, 1992). This idea is based on the distributional hypothe- sis (which states that “items with similar distributions have similar meanings”) to discover non-trivial patterns and re- lationships between data instances (Harris, 1954). The challenge of melding different data domains into a sin- gle conceptual space is the goal of multimodal representa- tions. A multimodal representation model is a function that maps objects that have multiple representations to a single vectorial space, combining the semantic implications that are expressed by distance similarities in unimodal repre- sentations (Atrey et al., 2010). Using a multimodal model, we can correlate words in a textual description of an object to the visual features of an image representing it, learning its “concept”. For example, a multimodal model could take as input the image of a sofa and return words that have sim- ilar meaning such as “chair” and “armchair”, or take one of these words as input and output a related image. The hypothesis of a multimodal model is that it can find fea- tures that are invariant to information modality, intrinsic to the concept, and can be subsequently used in any machine learning task (e.g. e-commerce product classification). …

Content truncated to 500 words