Mutual Information Maximization for Simple and Accurate Part-Of-Speech Induction

Proceedings of NAACL-HLT 2019, pages 1095–1104 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1095 Mutual Information Maximization for Simple and Accurate Part-Of-Speech Induction Karl Stratos Toyota Technological Institute at Chicago stratos@ttic.edu Abstract We address part-of-speech (POS) induction by maximizing the mutual information between the induced label and its context. We focus on two training objectives that are amenable to stochastic gradient descent (SGD): a novel generalization of the classical Brown cluster- ing objective and a recently proposed varia- tional lower bound. While both objectives are subject to noise in gradient updates, we show through analysis and experiments that the vari- ational lower bound is robust whereas the gen- eralized Brown objective is vulnerable. We obtain strong performance on a multitude of datasets and languages with a simple architec- ture that encodes morphology and context. 1 Introduction We consider information theoretic objectives for POS induction, an important unsupervised learning problem in computational linguistics (Christodoulopoulos et al., 2010). The idea is to make the induced label syntactically informa- tive by maximizing its mutual information with respect to local context. Mutual information has long been a workhorse in the development of NLP techniques, for instance the classical Brown clus- tering algorithm (Brown et al., 1992). But its role in today’s deep learning paradigm is less clear and a subject of active investigation (Belghazi et al., 2018; Oord et al., 2018). We focus on fully differentiable objectives that can be plugged into an automatic differentiation system and efficiently optimized by SGD. Specif- ically, we investigate two training objectives. The first is a novel generalization of the Brown cluster- ing objective obtained by relaxing the hard clus- tering constraint. The second is a recently pro- posed variational lower bound on mutual informa- tion (McAllester, 2018). A main challenge in optimizing these objectives is the difficulty of stochastic optimization. Each objective involves entropy estimation which is a nonlinear function of all data and does not de- compose over individual instances. This makes the gradients estimated on minibatches inconsis- tent with the true gradient estimated from the en- tire dataset. To our surprise, in practice we are able to optimize the variational objective effectively but not the generalized Brown objective. We analyze the estimated gradients and show that the incon- sistency error is only logarithmic in the former but linear in the latter. We validate our approach on POS induction by attaining strong performance on a multitude of datasets and languages. Our simple architecture that encodes morphology and context reaches up to 80.1 many-to-one accuracy on the 45-tag Penn WSJ dataset and achieves 4.7% absolute improve- ment to the previous best result on the universal treebank. Unlike previous works, our model does not rely on computationally expensive structured inference or hand-crafted features. 2 Background 2.1 Information Theory Mutual information. Mutual information be- tween two random variables measures the amount of information gained about one variable by ob- serving the other. Unlike the Pearson correlation coefficient which only captures the degree of …

Content truncated to 500 words