Big Data Small Data, In Domain Out-of Domain, Known Word Unknown Word: The Impact of Word Representations on Sequence Labelling Tasks

Proceedings of the 19th Conference on Computational Language Learning, pages 83–93, Beijing, China, July 30-31, 2015. c⃝2015 Association for Computational Linguistics Big Data Small Data, In Domain Out-of Domain, Known Word Unknown Word: The Impact of Word Representations on Sequence Labelling Tasks Lizhen Qu1,2, Gabriela Ferraro1,2, Liyuan Zhou1, Weiwei Hou1, Nathan Schneider3 and Timothy Baldwin1,4 1 NICTA, Australia 2 The Australian National University 3 University of Edinburgh 4 The University of Melbourne {lizhen.qu,gabriela.ferraro,liyuan.zho,weiwei.hou}@nicta.com.au nschneid@cs.cmu.edu tb@ldwin.net Abstract Word embeddings — distributed word representations that can be learned from unlabelled data — have been shown to have high utility in many natural language processing applications. In this paper, we perform an extrinsic evaluation of four popular word embedding methods in the context of four sequence labelling tasks: part-of-speech tagging, syntactic chunk- ing, named entity recognition, and multi- word expression identification. A particu- lar focus of the paper is analysing the ef- fects of task-based updating of word rep- resentations. We show that when using word embeddings as features, as few as several hundred training instances are suf- ficient to achieve competitive results, and that word embeddings lead to improve- ments over out-of-vocabulary words and also out of domain. Perhaps more sur- prisingly, our results indicate there is little difference between the different word em- bedding methods, and that simple Brown clusters are often competitive with word embeddings across all tasks we consider. 1 Introduction Recently, distributed word representations have grown to become a mainstay of natural language processing (NLP), and have been shown to have empirical utility in a myriad of tasks (Collobert and Weston, 2008; Turian et al., 2010; Baroni et al., 2014; Andreas and Klein, 2014). The un- derlying idea behind distributed word representa- tions is simple: to map each word w in vocabu- lary V onto a continuous-valued vector of dimen- sionality d ≪ |V |. Words that are similar (e.g., with respect to syntax or lexical semantics) will ideally be mapped to similar regions of the vec- tor space, implicitly supporting both generalisa- tion across in-vocabulary (IV) items, and counter- ing the effects of data sparsity for low-frequency and out-of-vocabulary (OOV) items. Without some means of automatically deriv- ing the vector representations without reliance on labelled data, however, word embeddings would have little practical utility. Fortunately, it has been shown that they can be “pre-trained” from unlabelled text data using various algorithms to model the distributional hypothesis (i.e., that words which occur in similar contexts tend to be semantically similar). Pre-training methods have been refined considerably in recent years, and scaled up to increasingly large corpora. As with other machine learning methods, it is well known that the quality of the pre-trained word embeddings depends heavily on factors including parameter optimisation, the size of the training data, and the fit with the target application. For example, Turian et al. (2010) showed that the op- timal dimensionality for word embeddings is task- specific. One factor which has received relatively little attention in NLP is the effect of “updating” the …

Content truncated to 500 words