The hunvec framework for NN-CRF-based sequential tagging

The Hunvec Framework For NN-CRF-based Sequential Tagging Katalin Pajkossy Attila Zs´eder BUTE, Institute of Mathematics HAS Research Institute for Linguistics H-1111 Egry J´ozsef u. 13-17, Budapest H-1068 Bencz´ur u 33, Budapest pajkossy@mokk.bme.hu zseder@mokk.bme.hu Abstract In this work we present the open source hunvec framework for sequential tagging, built upon Theano and Pylearn2. The underlying statistical model, which connects linear CRF-s with neural networks, was used by Collobert and co-workers, and several other researchers. For demonstrating the flexibility of our tool, we describe a set of experiments on part-of-speech and named-entity- recognition tasks, using English and Hungarian datasets, where we modify both model and training parameters, and illustrate the usage of custom features. Model parameters we experiment with affect the vectorial word representations used by the model; we apply different word vector initializations, defined by Word2vec and GloVe embeddings and enrich the representation of words by vectors assigned trigram features. We extend training methods by using their regularized (l2 and dropout) version. When testing our framework on a Hungarian named entity corpus, we find that its performance reaches the best published results on this dataset, with no need for language-specific feature engineering. Our code is available at http://github.com/zseder/hunvec Keywords: neural networks, sequential tagging, named entity recognition 1. Introduction and related work Sequential labeling models provide algorithms for NLP tasks of basic importance, including part-of-speech tagging (POS) and named entity recognition (NER). In this work we use a model which connects linear CRF-s with neural net- works, introduced in (Collobert and Weston, 2008; Peng et al., 2009; Do and Artieres, 2010). Our implementation is based on (Collobert et al., 2011). This work attracted much attention among NLP researchers, inspiring several adapta- tions (see (Zheng et al., 2013)) and extensions (see (Santos and Zadrozny, 2014)). The model is capable of utilizing distributed word representations as part of the model’s ini- tialization; the effect of using different embeddings has also been studied ((Demir and Ozgur, 2014), (Ling et al., 2015)). Although the results of the previous work are impressive, the number of tools publicly available for researchers is limited. For creating word embeddings there are excellent open source packages, Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014). There are also sev- eral low-level libraries for working with deep neural net- works, like Theano (Bergstra et al., 2010; Bastien et al., 2012), Torch (Collobert et al., 2002), or the recently pub- lished Tensorflow (Abadi et al., 2015). However, working with these frameworks requires solid programming skills and to the best of our knowledge, there is no publicly avail- able, high-level library for sequential tagging, that would let a broader range of researchers investigate the domain. Our framework, hunvec, built upon Theano, attempts to fill this gap. 2. Model and architecture Our model, following the work of (Collobert et al., 2011), uses their sentence-level training scheme. The log likeli- hood of a sentence s1s2 . . . sT having labels indexed by t1t2 . . . tT is defined as T � i=1 …

Content truncated to 500 words