SCARE ― The Sentiment Corpus of App Reviews with Fine-grained Annotations in German

SCARE – The Sentiment Corpus of App Reviews with Fine-grained Annotations in German Mario Sänger∗, Ulf Leser∗, Steffen Kemmerer§, Peter Adolphs§, Roman Klinger‡ ∗ Department of Computer Science, Humboldt Universität zu Berlin, Rudower Chaussee 25, 12489 Berlin, Germany § Neofonie GmbH, Robert-Koch-Platz 4, 10115 Berlin, Germany ‡ Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart, Pfaffenwaldring 5b, 70569 Stuttgart, Germany {saengerm,leser}@informatik.hu-berlin.de, {steffen.kemmerer,peter.adolphs}@neofonie.de, roman.klinger@ims.uni-stuttgart.de Abstract The automatic analysis of texts containing opinions of users about, e.g., products or political views has gained attention within the last decades. However, previous work on the task of analyzing user reviews about mobile applications in app stores is limited. Publicly available corpora do not exist, such that a comparison of different methods and models is difficult. We fill this gap by contributing the Sentiment Corpus of App Reviews (SCARE), which contains fine-grained annotations of application aspects, subjective (evaluative) phrases and relations between both. This corpus consists of 1,760 annotated application reviews from the Google Play Store with 2,487 aspects and 3,959 subjective phrases. We describe the process and methodology how the corpus was created. The Fleiss-κ between four annotators reveals an agreement of 0.72. We provide a strong baseline with a linear-chain conditional random field and word-embedding features with a performance of 0.62 for aspect detection and 0.63 for the extraction of subjective phrases. The corpus is available to the research community to support the development of sentiment analysis methods on mobile application reviews. Keywords: sentiment analysis, app reviews, German corpus, conditional random field, word embedding 1. Introduction Mobile devices, such as smartphones and tablets, are widespread in our society. Applications for these devices (also known as apps) become increasingly popular and gain a lot of attention in our daily lives. Applications are typically downloaded via app stores, i.e., application distribution plat- forms, such as the Apple App Store1, Google Play Store2, BlackBerry World3 or the Windows Store4. These platforms offer their users the possibility to assess applications with a 5-star rating and a textual review. An example for such a review is depicted in Figure 1. These reviews form a rich resource of information for app developers, since they hold the user’s opinions about the application itself and important aspects, like design and usability. Moreover, the reviews often contain complaints about problems and errors of the application as well as feature requests. Incorporating this feedback in the devel- opment process may have influence on the success of the app. However, one challenge for the developers is to deal with the overwhelming amount of reviews. Applications can have hundreds of thousands or even millions of reviews. A manual inspection and analysis of all these reviews is very time consuming and impractical. The app stores themselves offer only basic analytical capabilities. The analysis of opinions in reviews has been widely inves- tigated within the last decade (Pang and Lee, 2008) and is 1https://itunes.apple.com/us/genre/ios/ id36?mt=8 2https://play.google.com/store/ 3https://appworld.blackberry.com/ webstore/ 4https://www.microsoft.com/en-us/windows/ apps-and-games typically referred to as sentiment analysis or opinion min- ing (Liu, 2012). However, previous work in the area of …

Content truncated to 500 words