Imagination-Augmented Natural Language Understanding

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4392 - 4402 July 10-15, 2022 ©2022 Association for Computational Linguistics Imagination-Augmented Natural Language Understanding Yujie Lu1, Wanrong Zhu1, Xin Eric Wang2, Miguel Eckstein1, William Yang Wang1 1University of California, Santa Barbara, CA, USA {yujielu,wanrongzhu,migueleckstein,wangwilliamyang}@ucsb.edu 2University of California, Santa Cruz, CA, USA xwang366@ucsc.edu Abstract Human brains integrate linguistic and percep- tual information simultaneously to understand natural language, and hold the critical abil- ity to render imaginations. Such abilities en- able us to construct new abstract concepts or concrete objects, and are essential in in- volving practical knowledge to solve prob- lems in low-resource scenarios. However, most existing methods for Natural Language Understanding (NLU) are mainly focused on textual signals. They do not simulate hu- man visual imagination ability, which hin- ders models from inferring and learning ef- ficiently from limited data samples. There- fore, we introduce an Imagination-Augmented Cross-modal Encoder (iACE) to solve natu- ral language understanding tasks from a novel learning perspective—imagination-augmented cross-modal understanding. iACE enables vi- sual imagination with external knowledge trans- ferred from the powerful generative and pre- trained vision-and-language models. Extensive experiments on GLUE (Wang et al., 2018) and SWAG (Zellers et al., 2018) show that iACE achieves consistent improvement over visually- supervised pre-trained models. More impor- tantly, results in extreme and normal few-shot settings validate the effectiveness of iACE in low-resource natural language understanding circumstances.1 1 Introduction Cognitive neuroscience studies reveal neural acti- vation in vision-related brain areas when reading text (Just et al., 2004) and show a tight relationship between brain areas processing linguistic and vi- sual semantic information (Popham et al., 2021). In addition, visual imagery improves comprehen- sion during human language processing (Sadoski and Paivio, 1994). Such imagination empowers hu- 1Source code and pre-trained models are publicly available at https://github.com/YujieLu10/IACE-NLU Premise: A senior is waiting at the window of a restaurant that serves sandwiches. Hypothesis: A man is waiting in line for the bus. ? Contradiction? Entailment? Neutral? Figure 1: Rendering visual imagination is an intuitive way to activate perception for linguistic understanding, e.g. natural language inference. man brains with generalization capability to solve problems with limited supervision or data samples. However, the field of Natural language Under- standing has mainly been focused on building ma- chines based solely on language, ignoring the inher- ently grounded imagination from the external vi- sual world. These studies either learn text-only rep- resentations from language corpora (Devlin et al., 2019; Zhuang et al., 2021; Lan et al., 2020) or im- plicitly involve retrieved visual supervision in pre- trained language models (Tan and Bansal, 2020). Thus, their approaches appear limited in transfer- ring the connection between language understand- ing and visual imagination to downstream tasks, which are essential to solving low-resource circum- stances. In addition, these methods are limited to text-only augmentations, whereas visual imagina- tions leverage cross-modal augmentations to deal with low-resource situations. Human brains are multi-modal, integrating lin- guistic and perceptual information simultaneously. Intuitively, …

Content truncated to 500 words