Learning Architectures from an Extended Search Space for Language Modeling

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6629–6639 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6629 Learning Architectures from an Extended Search Space for Language Modeling Yinqiao Li1, Chi Hu1, Yuhao Zhang1, Nuo Xu1, Yufan Jiang1, Tong Xiao1,2∗, Jingbo Zhu1,2, Tongran Liu3, Changliang Li4 1NLP Lab, Northeastern University, Shenyang, China 2NiuTrans Research, Shenyang, China 3CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS, Beijing, China 4Kingsoft AI Lab, Beijing, China li.yin.qiao.2012@hotmail.com, {huchinlp,yoohao.zhang}@gmail.com, {xunuo0629,jiangyufan2018}@outlook.com, {xiaotong,zhujingbo}@mail.neu.edu.com, liutr@psych.ac.cn,lichangliang@kingsoft.com Abstract Neural architecture search (NAS) has ad- vanced significantly in recent years but most NAS systems restrict search to learning archi- tectures of a recurrent or convolutional cell. In this paper, we extend the search space of NAS. In particular, we present a general approach to learn both intra-cell and inter-cell architec- tures (call it ESS). For a better search result, we design a joint learning method to perform intra-cell and inter-cell NAS simultaneously. We implement our model in a differentiable architecture search system. For recurrent neu- ral language modeling, it outperforms a strong baseline significantly on the PTB and Wiki- Text data, with a new state-of-the-art on PTB. Moreover, the learned architectures show good transferability to other systems. E.g., they im- prove state-of-the-art systems on the CoNLL and WNUT named entity recognition (NER) tasks and CoNLL chunking task, indicating a promising line of research on large-scale pre- learned architectures. 1 Introduction Neural models have shown remarkable perfor- mance improvements in a wide range of natural language processing (NLP) tasks. Systems of this kind can broadly be characterized as following a neural network design: we model the problem via a pre-defined neural architecture, and the resulting network is treated as a black-box family of func- tions for which we find parameters that can general- ize well on test data. This paradigm leads to many successful NLP systems based on well-designed architectures. The earliest of these makes use of re- current neural networks (RNNs) for representation learning (Bahdanau et al., 2015; Wu et al., 2016), ∗Corresponding author. whereas recent systems have successfully incorpo- rated fully attentive models into language genera- tion and understanding (Vaswani et al., 2017). In designing such models, careful engineering of the architecture plays a key role for the state-of- the-art though it is in general extremely difficult to find a good network structure. The next obvi- ous step is toward automatic architecture design. A popular method to do this is neural architecture search (NAS). In NAS, the common practice is that we first define a search space of neural networks, and then find the most promising candidate in the space by some criteria. Previous efforts to make NAS more accurate have focused on improving search and network evaluation algorithms. But the search space is still restricted to a particular scope of neural networks. For example, most NAS meth- ods are applied to learn the topology in a recurrent or convolutional cell, but the connections between cells are still made in a heuristic …

Content truncated to 500 words