Quantifying the morphosyntactic content of Brown Clusters

Proceedings of NAACL-HLT 2019, pages 1541–1550 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1541 Quantifying the morphosyntactic content of Brown Clusters Manuel R. Ciosici UNSILO A/S and Aarhus University Aarhus, Denmark manuel@cs.au.dk Leon Derczynski IT University of Copenhagen Copenhagen, Denmark leod@itu.dk Ira Assent Department of Computer Science Aarhus University Aarhus, Denmark ira@cs.au.dk Abstract Brown and Exchange word clusters have long been successfully used as word representations in Natural Language Processing (NLP) sys- tems. Their success has been attributed to their seeming ability to represent both seman- tic and syntactic information. Using corpora representing several language families, we test the hypothesis that Brown and Exchange word clusters are highly effective at encoding mor- phosyntactic information. Our experiments show that word clusters are highly capable of distinguishing Parts of Speech. We show that increases in Average Mutual Information, the clustering algorithms’ optimization goal, are highly correlated with improvements in encod- ing of morphosyntactic information. Our re- sults provide empirical evidence that down- stream NLP systems addressing tasks depen- dent on morphosyntactic information can ben- efit from word cluster features. 1 Introduction Distributionally generated word classes (often re- ferred to as word clusters) are hard clusters, con- taining all word types observed in a corpus, allo- cated to clusters based on contextual information observed in the corpus. They have found wide use in Natural Language Processing (NLP) sys- tems as an alternative to word embeddings such as word2vec (Mikolov et al., 2013). Word clus- ters differentiate themselves from word embed- dings by requiring estimation of many fewer pa- rameters, and by their ability to derive qualitative representations from smaller corpora (Qu et al., 2015; Bansal et al., 2014). Brown Clusters (Brown et al., 1992) are a well- known approach based on hard, hierarchical, dis- tributionally derived groups of word types ob- served in a corpus of unstructured text, with Av- erage Mutual Information (AMI) as the optimiza- tion goal. Exchange Clusters are an alternative ap- proach obtained by applying the Exchange Algo- rithm (Kneser and Ney, 1993) to the same opti- mization goal. Unlike Brown, Exchange outputs a flat clustering, with no hierarchy (Martin et al., 1998). When only the bottom of the hierarchy is used, like in this paper, Exchange and Brown clus- ters are interchangeable. Both Brown and Exchange clusters have been used as word representations for various Natu- ral Language Processing tasks such as Part of Speech tagging in clean and noisy text (Swain and Cole, 2016; Owoputi et al., 2013; Derczyn- ski et al., 2015), dependency parsing (Koo et al., 2008; Bansal et al., 2014), Chinese Word Segmen- tation (Liang, 2005), and Named Entity Recog- nition (Swain and Cole, 2016; Derczynski et al., 2015; Liang, 2005). Word clusters distinguish themselves from word embedding models by their ability to learn from little data (Bansal et al., 2014; Qu et al., 2015); for example, in cases like (Bansal et al., 2014), word clusters outperform other kinds of representations, including word embeddings. In the literature, it …

Content truncated to 500 words