Distributional Representations for Handling Sparsity in Supervised Sequence-Labeling

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 495–503, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Distributional Representations for Handling Sparsity in Supervised Sequence-Labeling Fei Huang Temple University 1805 N. Broad St. Wachman Hall 324 tub58431@temple.edu Alexander Yates Temple University 1805 N. Broad St. Wachman Hall 324 yates@temple.edu Abstract Supervised sequence-labeling systems in natural language processing often suffer from data sparsity because they use word types as features in their prediction tasks. Consequently, they have difficulty estimat- ing parameters for types which appear in the test set, but seldom (or never) ap- pear in the training set. We demonstrate that distributional representations of word types, trained on unannotated text, can be used to improve performance on rare words. We incorporate aspects of these representations into the feature space of our sequence-labeling systems. In an ex- periment on a standard chunking dataset, our best technique improves a chunker from 0.76 F1 to 0.86 F1 on chunks begin- ning with rare words. On the same dataset, it improves our part-of-speech tagger from 74% to 80% accuracy on rare words. Fur- thermore, our system improves signifi- cantly over a baseline system when ap- plied to text from a different domain, and it reduces the sample complexity of se- quence labeling. 1 Introduction Data sparsity and high dimensionality are the twin curses of statistical natural language processing (NLP). In many traditional supervised NLP sys- tems, the feature space includes dimensions for each word type in the data, or perhaps even combi- nations of word types. Since vocabularies can be extremely large, this leads to an explosion in the number of parameters. To make matters worse, language is Zipf-distributed, so that a large frac- tion of any training data set will be hapax legom- ena, very many word types will appear only a few times, and many word types will be left out of the training set altogether. As a consequence, for many word types supervised NLP systems have very few, or even zero, labeled examples from which to estimate parameters. The negative effects of data sparsity have been well-documented in the NLP literature. The per- formance of state-of-the-art, supervised NLP sys- tems like part-of-speech (POS) taggers degrades significantly on words that do not appear in the training data, or out-of-vocabulary (OOV) words (Lafferty et al., 2001). Performance also degrades when the domain of the test set differs from the do- main of the training set, in part because the test set includes more OOV words and words that appear only a few times in the training set (henceforth, rare words) (Blitzer et al., 2006; Daum´e III and Marcu, 2006; Chelba and Acero, 2004). We investigate the use of distributional repre- sentations, which model the probability distribu- tion of a word’s context, as techniques for find- ing smoothed representations of word sequences. That is, we use the distributional representations to share information across unannotated examples of the same word type. We then compute features …

Content truncated to 500 words