Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3585–3590, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3585 Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition Yufan Jiang1, Chi Hu1, Tong Xiao12∗, Chunliang Zhang12, Jingbo Zhu12 1NLP Lab, Northeastern University, Shenyang, China 2NiuTrans Research, Shenyang, China jiangyufan2018@outlook.com, huchinlp@gmail.com {xiaotong, zhangchunliang, zhujingbo}@mail.neu.edu.cn Abstract In this paper, we study differentiable neural architecture search (NAS) methods for natu- ral language processing. In particular, we im- prove differentiable architecture search by re- moving the softmax-local constraint. Also, we apply differentiable NAS to named entity recognition (NER). It is the first time that dif- ferentiable NAS methods are adopted in NLP tasks other than language modeling. On both the PTB language modeling and CoNLL-2003 English NER data, our method outperforms strong baselines. It achieves a new state-of- the-art on the NER task. 1 Introduction Neural architecture search (NAS) has become popular recently in machine learning for their abil- ity to find new models and to free researchers from the hard work of designing network architectures. The earliest of these approaches use reinforcemen- t learning (RL) to learn promising architectures in a discrete space (Zoph and Le, 2016), where- as others have successfully modeled the problem in a continuous manner (Liu et al., 2019; Xie et al., 2019b; Huang and Xiang, 2019). As an instance of the latter, differentiable architecture search (DARTS) employs continuous relaxation to architecture representation and makes gradient de- scent straightforwardly applicable to search. This leads to an efficient search process that is orders of magnitude faster than the RL-based counterparts. Like recent methods in NAS (Xie and Yuille, 2017; Zoph and Le, 2016; Baker et al., 2016), DARTS represents networks as a directed acyclic graph for a given computation cell (see Figure 1(a)). An edge between nodes performs a pre- defined operation to transform the input (i.e., tail) ∗Corresponding author. xt ht−1 0 1 2 3 s s s s s s s : softmax (a) DARTS cell xt ht−1 0 1 2 3 s s s (b) Our cell Figure 1: An overview of DARTS cell and our cell to the output (i.e., head). For a continuous net- work space, DARTS uses the softmax trick to re- lax the categorical choice of edges to soft deci- sions. Then, one can optimize over the graph us- ing standard gradient descent methods. The opti- mized network is inferred by choosing the edges with maximum weights in softmax. However, DARTS is a “local” model because the softmax-based relaxation is imposed on each bundle of edges between two nodes. This leads to a biased model in that edges coming from different nodes are not comparable. Such a constraint limit- s the inference space to sub-graphs with one edge between each pair of nodes. Also, the learned net- work might be redundant because every node has to receive edges from all predecessors no matter they …

Content truncated to 500 words