Relational Lasso —An Improved Method Using the Relations Among Features—

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 1080–1088, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Relational Lasso —An Improved Method Using the Relations among Features— Kotaro Kitagawa Kumiko Tanaka-Ishii Graduate School of Information Science and Technology, The University of Tokyo kitagawa@cl.ci.i.u-tokyo.ac.jp kumiko@i.u-tokyo.ac.jp Abstract Relational lasso is a method that incor- porates feature relations within machine learning. By using automatically obtained noisy relations among features, relational lasso learns an additional penalty parame- ter per feature, which is then incorporated in terms of a regularizer within the target optimization function. Relational lasso has been tested on three different tasks: text categorization, po- larity estimation, and parsing, where it was compared with conventional lasso and adaptive lasso (Zou, 2006) when using a multi-class logistic regression optimiza- tion method. Relational lasso outper- formed these other lasso methods in the tests. 1 Introduction As machine learning methods scale up and now deal with millions of features, we ideally want to add all possible features without having to man- ually verify or consider the effectiveness of each feature with respect to the performance. In other words, we need an automatic way to exploit any usable information that can be obtained from fea- tures. However, with current machine learning methods, adding noisy features could lower per- formance, so the user still has to decide which fea- tures are worth adding. Regularization methods have recently received greater interest because of this need. Regular- ization is expressed as a constraint term within an optimization function, where the term is given as a function regarding the importance weight of each feature. Regularization provides a means of importance control embedded within the target optimization problem. Among the various regu- larizers, lasso, proposed by (Tibshirani, 1996) is the most widely used because of its mathematical comprehensiveness. This paper describes a method, which we call relational lasso, that improves upon the conven- tional lasso method. We show that relational lasso improves the overall performance of classi- fication compared with that of other lasso meth- ods. This study was motivated through a limitation we observed in conventional lasso: that features could be inter-related, but such dependences are not incorporated within the current regularization. Therefore, the conventional method tends to favor correlated features, which can lead to the impor- tance of non-correlated features being neglected. Relational lasso overcomes this limitation of conventional lasso by introducing an additional penalty parameter for each feature; this parame- ter is estimated automatically given the noisy re- lations among features, where the relations are also automatically generated. While the proposed method does not add to the computational com- plexity of the conventional regularization method, it improves the quality of classification. We ex- plain the method and show empirical results from tests based on three different text classification and parsing tasks. Regularization was originally proposed as a way to avoid over-fitting by favoring some small number of features. Our method presented in this article exploits this approach further and attempts to estimate the …

Content truncated to 500 words