Learning a Product of Experts with Elitist Lasso

International Joint Conference on Natural Language Processing, pages 525–533, Nagoya, Japan, 14-18 October 2013. Learning a Product of Experts with Elitist Lasso Mengqiu Wang Computer Science Department Stanford University Stanford, CA 94305, USA mengqiu@cs.stanford.edu Christopher D. Manning Computer Science Department Stanford University Stanford, CA 94305, USA manning@cs.stanford.edu Abstract Discriminative models such as logistic re- gression profit from the ability to incorpo- rate arbitrary rich features; however, com- plex dependencies among overlapping fea- tures can often result in weight undertrain- ing. One popular method that attempts to mitigate this problem is logarithmic opin- ion pools (LOP), which is a specialized form of product of experts model that au- tomatically adjusts the weighting among experts. A major problem with LOP is that it requires significant amounts of do- main expertise in designing effective ex- perts. We propose a novel method that learns to induce experts — not just the weighting between them — through the use of a mixed ℓ2ℓ1 norm as previously seen in elitist lasso. Unlike its more popular sibling ℓ1ℓ2 norm (used in group lasso), which seeks feature sparsity at the group-level, ℓ2ℓ1 norm encourages spar- sity within feature groups. We demon- strate how this property can be leveraged as a competition mechanism to induce groups of diverse experts, and introduce a new formulation of elitist lasso Max- Ent in the FOBOS optimization frame- work (Duchi and Singer, 2009). Results on Named Entity Recognition task suggest that this method gives consistent improve- ments over a standard logistic regression model, and is more effective than conven- tional induction schemes for experts. 1 Introduction Conditionally trained discriminative models like logistic regression (a.k.a., MaxEnt models) gain power from their ability to incorporate a large number of arbitrarily overlapping features. But such models also exhibit complex feature depen- dencies and therefore are susceptible to the prob- lem of weight undertraining — the contributions of certain features are overlooked because of fea- tures they co-occur with (Sutton et al., 2007; Hin- ton et al., 2012). One popular method that attempts to mitigate this problem is logarithmic opinion pools (LOP) (Heskes, 1998; Smith et al., 2005; Sutton et al., 2007). LOP works in a similar fashion to a prod- uct of experts model, in which a number of experts are individually assembled first, and then their pre- dictions are combined multiplicatively to create an ensemble model. Experts are generated by first partitioning the feature space into subsets, then an independent model (expert) is trained on each sub- set of features (Sutton et al., 2007). Under the assumption that strongly correlated features are partitioned into separate subsets, this method ef- fectively forces the models to learn how to make predictions with each subset of the features inde- pendently. It also helps in scenarios where some strong features stop other features from getting weight, a problem known as feature co-adaptation. The theoretical justification for favoring a product ensemble over an additive ensemble is that infer- ence with a product of log-linear models is sig- nificantly easier than …

Content truncated to 500 words