Does Higher Order LSTM Have Better Accuracy for Segmenting and Labeling Sequence Data?

Proceedings of the 27th International Conference on Computational Linguistics, pages 723–733 Santa Fe, New Mexico, USA, August 20-26, 2018. 723 Does Higher Order LSTM Have Better Accuracy for Segmenting and Labeling Sequence Data? Yi Zhang, Xu Sun, Shuming Ma, Yang Yang, Xuancheng Ren MOE Key Lab of Computational Linguistics, School of EECS, Peking University {zhangyi16, xusun, shumingma, 1200012760, renxc}@pku.edu.cn Abstract Existing neural models usually predict the tag of the current token independent of the neigh- boring tags. The popular LSTM-CRF model considers the tag dependencies between every two consecutive tags. However, it is hard for existing neural models to take longer distance dependen- cies of tags into consideration. The scalability is mainly limited by the complex model structures and the cost of dynamic programming during training. In our work, we first design a new model called “high order LSTM” to predict multiple tags for the current token which contains not only the current tag but also the previous several tags. We call the number of tags in one prediction as “order”. Then we propose a new method called Multi-Order BiLSTM (MO-BiLSTM) which combines low order and high order LSTMs together. MO-BiLSTM keeps the scalability to high order models with a pruning technique. We evaluate MO-BiLSTM on all-phrase chunking and NER datasets. Experiment results show that MO-BiLSTM achieves the state-of-the-art result in chunking and highly competitive results in two NER datasets. 1 1 Introduction Chunking and named entity recognition are sequence labeling tasks whose target is to find the correct segments and give them the correct labels. The tags inside a segment have internal dependencies. The tags in consecutive segments may have dependencies, too. Therefore, it is natural to take the tag depen- dencies into consideration when making a prediction in such sequence labeling tasks. Recently, methods have been proposed to capture tag dependencies for neural networks. Collobert et al. (2011) proposed a method based on convolutional neural networks, which can use dynamic program- ming in training and testing stage (like a CRF layer) to capture tag dependencies. Furthermore, Huang et al. (2015) proposed LSTM-CRF by combining LSTM and CRF for structured learning. They use a transition matrix to model the tag dependencies. A similar structure is adopted by Ma and Hovy (2016). Their model also involves an external layer to extract some character level features. However, it is not explicit how to model the dependencies of more tags or use the dependency infor- mation in these lines of work. We then propose a solution to capture long distance tag dependencies and use them for dependency-aware prediction of tags. For clarity, we first give some detailed explanations of the related terms in our work. “order” means the number of tags that a prediction involves in a model. An order-2 tag is a bigram which contains the previous tag and the current tag at a certain time step, as shown in Figure 1. Higher order tags are defined in a similar way. We first develop a simple method to implement high order …

Content truncated to 500 words