Character-Level Feature Extraction with Densely Connected Networks

Proceedings of the 27th International Conference on Computational Linguistics, pages 3228–3239 Santa Fe, New Mexico, USA, August 20-26, 2018. 3228 Character-Level Feature Extraction with Densely Connected Networks Chanhee Lee1, Young-Bum Kim2, Dongyub Lee1, HeuiSeok Lim1∗ 1Korea University, Republic of Korea {chanhee0222, judelee93, limhseok}@korea.ac.kr 2Amazon Alexa youngbum@amazon.com Abstract Generating character-level features is an important step for achieving good results in various nat- ural language processing tasks. To alleviate the need for human labor in generating hand-crafted features, methods that utilize neural architectures such as Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN) to automatically extract such features have been proposed and have shown great results. However, CNN generates position-independent features, and RNN is slow since it needs to process the characters sequentially. In this paper, we propose a novel method of using a densely connected network to automatically extract character-level features. The proposed method does not require any language or task specific assumptions, and shows robustness and effectiveness while being faster than CNN- or RNN-based methods. Evaluat- ing this method on three sequence labeling tasks - slot tagging, Part-of-Speech (POS) tagging, and Named-Entity Recognition (NER) - we obtain state-of-the-art performance with a 96.62 F1-score and 97.73% accuracy on slot tagging and POS tagging, respectively, and comparable performance to the state-of-the-art 91.13 F1-score on NER. 1 Introduction Effectively extracting character-level features from words is crucial in many Natural Language Process- ing (NLP) tasks, such as Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and Slot tagging. Thus, most state-of-the-art methods for these tasks exploit some kind of character-level fea- tures (Huang et al., 2015; dos Santos and Zadrozny, 2014). Recently, generating character-level features with neural architectures such as Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN) has drawn much attention, mainly because it doesn’t require human labor and shows superior performance (Ma and Hovy, 2016; dos Santos and Zadrozny, 2014). However, CNN struggles at distin- guishing anagrams, and RNN is inherently slow due to its sequential nature. In this paper, we propose an effective and efficient way of extracting character-level features using a densely connected network. The key benefits of the proposed method can be summarized as follows. First, it does not require any hand-crafted features or data preprocessing. Each word is processed based on n-gram statistics of the training data, and vectorized using bag-of-characters. Additional features are based on hexadecimal values of the character-set (e.g. UTF-16) and number of characters in the word. Second, it extracts effective character-level features while being efficient. State-of-the-art performance can be achieved using this method, and the feature extraction is done with a simple densely connected network with a single hidden layer. Third, it doesn’t depend on features that are language or task specific, such as character type features or gazetteer (i.e. lists of known named entities such as cities or organiza- tion names). The only requirement for adopting this method is that the language should be processable as a sequence of words, which is made of sequence of characters. These benefits, combined with minimum …

Content truncated to 500 words