Multi-Domain Learning: When Do Domains Matter?

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1302–1312, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Multi-Domain Learning: When Do Domains Matter? Mahesh Joshi School of Computer Science Carnegie Mellon University Pittsburgh, PA, 15213, USA maheshj@cs.cmu.edu Mark Dredze Human Language Technology Center of Excellence Johns Hopkins University Baltimore, Maryland 21211 mdredze@cs.jhu.edu William W. Cohen School of Computer Science Carnegie Mellon University Pittsburgh, PA, 15213, USA wcohen@cs.cmu.edu Carolyn P. Ros´e School of Computer Science Carnegie Mellon University Pittsburgh, PA, 15213, USA cprose@cs.cmu.edu Abstract We present a systematic analysis of exist- ing multi-domain learning approaches with re- spect to two questions. First, many multi- domain learning algorithms resemble ensem- ble learning algorithms. (1) Are multi-domain learning improvements the result of ensemble learning effects? Second, these algorithms are traditionally evaluated in a balanced class la- bel setting, although in practice many multi- domain settings have domain-specific class label biases. When multi-domain learning is applied to these settings, (2) are multi- domain methods improving because they cap- ture domain-specific class biases? An under- standing of these two issues presents a clearer idea about where the field has had success in multi-domain learning, and it suggests some important open questions for improving be- yond the current state of the art. 1 Introduction Research efforts in recent years have demonstrated the importance of domains in statistical natural lan- guage processing. A mismatch between training and test domains can negatively impact system accuracy as it violates a core assumption in many machine learning algorithms: that data points are indepen- dent and identically distributed (i.i.d.). As a result, numerous domain adaptation methods (Chelba and Acero, 2004; Daum´e III and Marcu, 2006; Blitzer et al., 2007) target settings with a training set from one domain and a test set from another. Often times the training set itself violates the i.i.d. assumption and contains multiple domains. In this case, training a single model obscures domain dis- tinctions, and separating the dataset by domains re- duces training data. Instead, multi-domain learn- ing (MDL) can take advantage of these domain la- bels to improve learning (Daum´e III, 2007; Dredze and Crammer, 2008; Arnold et al., 2008; Finkel and Manning, 2009; Zhang and Yeung, 2010; Saha et al., 2011). One such example is sentiment classification of product reviews. Training data is available from many product categories and while all data should be used to learn a model, there are important differ- ences between the categories (Blitzer et al., 2007)1. While much prior research has shown improve- ments using MDL, this paper explores what prop- erties of an MDL setting matter. Are previous im- provements from MDL algorithms discovering im- portant distinctions between features in different do- mains, as we would hope, or are other factors con- tributing to learning success? The key question of this paper is: when do domains matter? Towards this goal we explore two issues. First, we explore the question of whether domain distinc- tions …

Content truncated to 500 words