Hierarchically-Refined Label Attention Network for Sequence Labeling

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4115–4128, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4115 Hierarchically-Refined Label Attention Network for Sequence Labeling Leyang Cui1,2,3 and Yue Zhang2,3 1Zhejiang University 2School of Engineering, Westlake University 3Institute of Advanced Technology, Westlake Institute for Advanced Study cuileyang@westlake.edu.cn, yue.zhang@wias.org.cn Abstract CRF has been used as a powerful model for statistical sequence labeling. For neural se- quence labeling, however, BiLSTM-CRF does not always lead to better results compared with BiLSTM-softmax local classification. This can be because the simple Markov label tran- sition model of CRF does not give much in- formation gain over strong neural encoding. For better representing label sequences, we in- vestigate a hierarchically-refined label atten- tion network, which explicitly leverages label embeddings and captures potential long-term label dependency by giving each word incre- mentally refined label distributions with hier- archical attention. Results on POS tagging, NER and CCG supertagging show that the pro- posed model not only improves the overall tag- ging accuracy with similar number of parame- ters, but also significantly speeds up the train- ing and testing compared to BiLSTM-CRF. 1 Introduction Conditional random fields (CRF) (Lafferty et al., 2001) is a state-of-the-art model for statistical sequence labeling (Toutanova et al., 2003; Peng et al., 2004; Ratinov and Roth, 2009). Recently, CRF has been integrated with neural encoders as an output layer to capture label transition patterns (Zhou and Xu, 2015; Ma and Hovy, 2016). This, however, sees mixed results. For example, pre- vious work (Reimers and Gurevych, 2017; Yang et al., 2018) has shown that BiLSTM-softmax gives better accuracies compared to BiLSTM- CRF for part-of-speech (POS) tagging. In addi- tion, the state-of-the-art neural Combinatory Cat- egorial Grammar (CCG) supertaggers do not use CRF (Xu et al., 2015; Lewis et al., 2016). One possible reason is that the strong rep- resentation power of neural sentence encoders such as BiLSTMs allow models to capture im- plicit long-range label dependencies from input They can fish and also tomatoes here Input PRP 0.9 … … … … MD 0.7 VB 0.2 … … VB 0.5 NN 0.4 … … CC 0.9 … … … … RB 0.9 … … … … NN 0.9 … … … … RB 0.9 … … … … PRP 0.9 … … … … MD 0.3 VB 0.7 … … NN 0.8 VB 0.2 … … CC 0.9 … … … … RB 0.9 … … … … NN 0.9 … … … … RB 0.9 … … … … PRP VB NN CC RB NN RB First Layer Last Layer Output Figure 1: Visualization of hierarchically-refined Label Attention Network for POS tagging. The numbers in- dicate the label probability distribution for each word. word sequences alone (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2016; Teng and Zhang, 2018), thereby allowing the output layer to make local predictions. In contrast, though explicitly capturing output label dependencies, CRF …

Content truncated to 500 words