Online Methods for Multi-Domain Learning and Adaptation

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 689–697, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Online Methods for Multi-Domain Learning and Adaptation Mark Dredze and Koby Crammer Department of Computer and Information Science University of Pennsylvania Philadelphia, PA 19104 USA {mdredze,crammer}@cis.upenn.edu Abstract NLP tasks are often domain specific, yet sys- tems can learn behaviors across multiple do- mains. We develop a new multi-domain online learning framework based on parameter com- bination from multiple classifiers. Our algo- rithms draw from multi-task learning and do- main adaptation to adapt multiple source do- main classifiers to a new target domain, learn across multiple similar domains, and learn across a large number of disparate domains. We evaluate our algorithms on two popular NLP domain adaptation tasks: sentiment clas- sification and spam filtering. 1 Introduction Statistical classifiers routinely process millions of websites, emails, blogs and other text every day. Variability across different data sources means that training a single classifier obscures differences and separate classifiers ignore similarities. Similarly, adding new domains to existing systems requires adapting existing classifiers. We present new online algorithms for three multi- domain learning scenarios: adapting existing classi- fiers to new domains, learning across multiple simi- lar domains and scaling systems to many disparate domains. Multi-domain learning combines char- acteristics of both multi-task learning and domain adaptation and drawing from both areas, we de- velop a multi-classifier parameter combination tech- nique for confidence-weighted (CW) linear classi- fiers (Dredze et al., 2008). We focus on online algo- rithms that scale to large amounts of data. Next, we describe multi-domain learning and re- view the CW algorithm. We then consider our three settings using multi-classifier parameter combina- tion. We conclude with related work. 2 Multi-Domain Learning In online multi-domain learning, each instance x is drawn from a domain d specific distribution x ∼ Dd over a vectors space RN and labeled with a domain specific function fd with label y ∈ {−1, +1} (for binary classification.) On round i the classifier re- ceives instance xi and domain identifier di and pre- dicts label ˆyi ∈ {−1, +1}. It then receives the true label yi ∈ {−1, +1} and updates its prediction rule. As an example, consider a multi-user spam fil- ter, which must give high quality predictions for new users (without new user data), learn on multi- ple users simultaneously and scale to thousands of accounts. While a single classifier trained on all users would generalize across users and extend to new users, it would fail to learn user-specific prefer- ences. Alternatively, separate classifiers would cap- ture user-specific behaviors but would not general- ize across users. The approach we take to solv- ing multi-domain problems is to combine domain- specific classifiers. In the adaptation setting, we combine source domain classifiers for a new tar- get domain. For learning across domains, we com- bine domain-specific classifiers and a shared classi- fier learned across all domains. For learning across disparate domains we learn which domain-specific and shared classifiers …

Content truncated to 500 words