Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6994–7007 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6994 Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions Tian Jin∗ IBM Thomas J. Watson Research Center Yorktown Heights, New York 10598 tian.jin2@ibm.com Zhun Liu∗♦ Microsoft Bellevue, WA 98004 zhunliu@microsoft.com Shengjia Yan Tandon School of Engineering New York University Brooklyn, NY 11201 sjyan@nyu.edu Alexandre Eichenberger IBM Thomas J. Watson Research Center Yorktown Heights, New York 10598 alexe@us.ibm.com Louis-Philippe Morency School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 morency@cs.cmu.edu Abstract Transfer learning using ImageNet pre-trained models has been the de facto approach in a wide range of computer vision tasks. However, fine-tuning still requires task-specific training data. In this paper, we propose N3 (Neural Networks from Natural Language) - a new paradigm of synthesizing task-specific neural networks from language descriptions and a generic pre-trained model. N3 leverages lan- guage descriptions to generate parameter adap- tations as well as a new task-specific classifica- tion layer for a pre-trained neural network, ef- fectively “fine-tuning” the network for a new task using only language descriptions as in- put. To the best of our knowledge, N3 is the first method to synthesize entire neural networks from natural language. Experimen- tal results show that N3 can out-perform pre- vious natural-language based zero-shot learn- ing methods across 4 different zero-shot image classification benchmarks. We also demon- strate a simple method to help identify key- words in language descriptions leveraged by N3 when synthesizing model parameters.1 1 Introduction A person with generic world knowledge can learn to perform a new task based on verbal instructions. On the other hand, despite recent successes in deep ∗First two authors contributed equally. ♦This work was performed when Zhun Liu was affiliated with Carnegie Mellon University. 1Code is released at https://github.com/tjingrant/n3cr . Husky This breed has a well rounded, apple like shaped head with a muzzle that is tiny in contrast to the head. Chiwawa Husky Golden Retriever N3 CNN to Classify Dogs Chihuahua Classify Synthesize Run Once Figure 1: An illustration of N3. A list of object class descriptions is fed to the N3 model to produce a CNN classifier that can classify images belonging to the ob- ject classes described. learning, it remains challenging to re-purpose pre- trained visual classification models to recognize a new set of objects without labeling a new training dataset. A natural question emerges from this ob- servation: can a computer also learn to recognize new objects, simply by reading the descriptions of them in natural language? Concretely, can we cre- ate visual classifiers using language descriptions of the objects of interest? In this paper, we introduce a new paradigm for synthesizing task-specific neural networks for im- age classification simply from language descrip- tions of the relevant objects. We propose N3 - Neural Networks from Natural Language, a meta- model that takes a list of object descriptions as input to produce a classification model for these objects, as …

Content truncated to 500 words