Design Challenges and Misconceptions in Neural Sequence Labeling

Proceedings of the 27th International Conference on Computational Linguistics, pages 3879–3889 Santa Fe, New Mexico, USA, August 20-26, 2018. 3879 Design Challenges and Misconceptions in Neural Sequence Labeling Jie Yang, Shuailong Liang, Yue Zhang Singapore University of Technology and Design {jie yang, shuailong liang}@mymail.sutd.edu.sg yue zhang@sutd.edu.sg Abstract We investigate the design challenges of constructing effective and efficient neural sequence la- beling systems, by reproducing twelve neural sequence labeling models, which include most of the state-of-the-art structures, and conduct a systematic model comparison on three benchmarks (i.e. NER, Chunking, and POS tagging). Misconceptions and inconsistent conclusions in ex- isting literature are examined and clarified under statistical experiments. In the comparison and analysis process, we reach several practical conclusions which can be useful to practitioners. 1 Introduction Sequence labeling models have been used for fundamental NLP tasks such as POS tagging, chunking and named entity recognition (NER). Traditional work uses statistical approaches such as Hidden Markov Models (HMM) and Conditional Random Fields (CRF) (Ratinov and Roth, 2009; Passos et al., 2014; Luo et al., 2015) with handcrafted features and task-specific resources. With advances in deep learning, neural models have given state-of-the-art results on many sequence labeling tasks (Ling et al., 2015; Lample et al., 2016; Ma and Hovy, 2016). Words and characters are encoded in distributed representations (Mikolov et al., 2013) and sentence-level features are learned automatically during end-to-end training. Many existing state-of-the-art neural sequence labeling models utilize word-level Long Short-Term Memory (LSTM) structures to represent global sequence information and a CRF layer to capture dependencies between neighboring labels (Huang et al., 2015; Lample et al., 2016; Ma and Hovy, 2016; Peters et al., 2017). As an alternative, Convolution Neural Network (CNN) (LeCun et al., 1989) has also been used for its ability of parallel computing, leading to an efficient training and decoding process. Despite them being dominant in the research literature, reproducing published results for neural mod- els can be challenging, even if the codes are available open source. For example, Reimers and Gurevych (2017b) conduct a large number of experiments using the code of Ma and Hovy (2016), but cannot obtain comparable results as reported in the paper. Liu et al. (2018) report lower average F-scores on NER when reproducing the structure of Lample et al. (2016), and on POS tagging when reproducing Ma and Hovy (2016). Most literature compares results with others by citing the scores directly (Huang et al., 2015; Lample et al., 2016) without re-implementing them under the same setting, resulting in less persuasiveness on the advantage of their models. In addition, conclusions from different reports can be contradictory. For example, most work observes that stochastic gradient descent (SGD) gives best performance on NER task (Chiu and Nichols, 2016; Lample et al., 2016; Ma and Hovy, 2016), while Reimers and Gurevych (2017b) report that SGD is the worst optimizer on the same datasets. The comparison between different deep neural models is challenging due to sensitivity on experimental •settings. We list six inconsistent configurations in literature, which lead …

Content truncated to 500 words