Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2066–2080, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2066 Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision Hao Tan Mohit Bansal UNC Chapel Hill {haotan, mbansal}@cs.unc.edu Abstract Humans learn language by listening, speak- ing, writing, reading, and also, via interaction with the multimodal real world. Existing lan- guage pre-training frameworks show the ef- fectiveness of text-only self-supervision while we explore the idea of a visually-supervised language model in this paper. We find that the main reason hindering this exploration is the large divergence in magnitude and distribu- tions between the visually-grounded language datasets and pure-language corpora. There- fore, we develop a technique named “vok- enization” that extrapolates multimodal align- ments to language-only data by contextually mapping language tokens to their related im- ages (which we call “vokens”). The “vo- kenizer” is trained on relatively small im- age captioning datasets and we then apply it to generate vokens for large language cor- pora. Trained with these contextually gener- ated vokens, our visually-supervised language models show consistent improvements over self-supervised alternatives on multiple pure- language tasks such as GLUE, SQuAD, and SWAG.1 1 Introduction Most humans learn language understanding from multiple modalities rather than only from the text and audio, especially using the visual modality. As claimed in Bloom (2002), visual pointing is an essential step for most children to learn mean- ings of words. However, existing language pre- training frameworks are driven by contextual learn- ing which only takes the language context as self- supervision. For example, word2vec (Mikolov et al., 2013) takes surrounding bag-of-words; ELMo (Peters et al., 2018) and GPT (Radford et al., 1Code and pre-trained models publicly available at: https://github.com/airsplay/vokenization. Vokens (Token-Related Images) Humans learn language by listening, speaking Language Tokens Visually- Supervised Language Model Visual Supervision Language Input …… Vokenization Figure 1: We visually supervise the language model with token-related images. We call these images vo- kens (visualized tokens) and develop a vokenization process to contextually generate them. 2018) take succeeding contexts; and BERT (De- vlin et al., 2019) takes randomly masked tokens. Although these self-supervised frameworks have achieved strong progress towards understanding human language, they did not borrow grounding information from the external visual world (see re- lated motivations in recent work by Bender and Koller (2020) and Bisk et al. (2020)). In this paper, we introduce the visually- supervised language model that simulates human language learning with visual pointing (Bloom, 2002). As shown in Fig. 1, this model takes lan- guage tokens as input and uses token-related im- ages as visual supervision. We name these images as vokens (i.e., visualized tokens), since they act as visualizations of the corresponding tokens. As- suming that a large aligned token-voken dataset exists, the model could learn from these vokens via voken-prediction tasks. Unfortunately, such an aligned token-voken dataset is currently unavailable and hence there are two main challenges in creating it from visually- grounded language datasets. First, there is a large discrepancy between visually-grounded …

Content truncated to 500 words