Revisiting the Old Kitchen Sink: Do we Need Sentiment Domain Adaptation?

Proceedings of Recent Advances in Natural Language Processing, pages 420–427, Hissar, Bulgaria, 7-13 September 2013. Revisiting The Old Kitchen Sink: Do We Need Sentiment Domain Adaptation? Riham Hassan Mansour1 rihamma@microsoft.com Nesma Refaei3 nesma.a.refaei@eng.cu.edu.eg Michael Gamon2 mgamon@microsoft.com Khaled Sami1 t-ksami@microsoft.com Ahmed Abdel-Hamid1 ahmedab@microsoft.com 1Microsoft Research, 306 Maadi Courniche, Basatin, Cairo, Egypt 2Microsoft Research, One Microsoft Way, Redmond, WA 98051, USA 3Computer Engineering, Cairo University/ Gamaet El Qahera St., Giza, Egypt Abstract In this paper we undertake a large cross- domain investigation of sentiment domain ad- aptation, challenging the practical necessity of sentiment domain adaptation algorithms. We first show that across a wide set of domains, a simple “all-in-one” classifier that utilizes all available training data from all but the target domain tends to outperform published domain adaptation methods. A very simple ensemble classifier also performs well in these scenarios. Combined with the fact that labeled data now- adays is inexpensive to come by, the “kitchen sink” approach, while technically non- glamorous, might be perfectly adequate in practice. We also show that the common anec- dotal evidence for sentiment terms that “flip” polarity across domains is not borne out em- pirically. 1 Introduction Automatic detection and analysis of sentiment around products, brands, political issues etc. has triggered a large amount of research in the past 15 – 20 years (for a recent overview see Pang & Lee 2008 and Liu 2012). Early work focused on algorithms for mining sentiment dictionaries (Hatzivassiloglou and McKeown 1997, Turney 2002); this was followed by the exploration of supervised techniques (Pang et al. 2002) and, somewhat more recently, by investigations of domain adaptation techniques. Also more recent- ly, the focus has broadened from the detection of polarity (negative/positive sentiment) to more nuanced approaches that try to identify targets and holders of sentiment, sentiment strength, or finer-grained mood distinctions (e.g. Wilson et al. 2006, Kim and Hovy 2006). Within the polar- ity detection paradigm, a number of common assumptions have been shared in the community and are frequently repeated in the literature. Two of these fundamental assumptions are: 1. Obtaining sufficient labeled data for su- pervised training is expensive 2. Sentiment models trained on one domain tend to perform poorly on new, unseen domains A conclusion that is often drawn from these assumptions is that domain adaptation of senti- ment models from a domain with sufficient la- beled data to a new domain with little labeled data is an important problem and requires new and sophisticated algorithms. In this paper, we empirically re-examine the assumptions above. Based on a wide range of experiments on 27 different domains, we chal- lenge the conclusion that domain adaptation for polarity detection necessarily requires novel and sophisticated machinery. It is important to keep in mind, however, that our claims are strictly limited to the problem under investigation, namely polarity detection. We do not make any claims whatsoever about domain adaptation for other sentiment-related problems or general problems in machine learning. Based on readily available data from 27 domains, we show that a …

Content truncated to 500 words