DRANZIERA: An Evaluation Protocol For Multi-Domain Opinion Mining

DRANZIERA: An Evaluation Protocol For Multi-Domain Opinion Mining Mauro Dragoni1, Andrea G. B. Tettamanzi2, C´elia da Costa Pereira2 1 FBK–IRST, Trento, Italy 2 Universit´e Nice Sophia Antipolis, I3S, UMR 7271, Sophia Antipolis, France dragoni@fbk.eu, andrea.tettamanzi@unice.fr, celia.pereira@unice.fr Abstract Opinion Mining is a topic which attracted a lot of interest in the last years. By observing the literature, it is often hard to replicate system evaluation due to the unavailability of the data used for the evaluation or to the lack of details about the protocol used in the campaign. In this paper, we propose an evaluation protocol, called DRANZIERA, composed of a multi-domain dataset and guidelines allowing both to evaluate opinion mining systems in different contexts (Closed, Semi-Open, and Open) and to compare them to each other and to a number of baselines. Keywords: Opinion Mining, Benchmark, Evaluation Protocol 1. Introduction Opinion Mining is a natural language processing task whose aim is to classify documents according to the opin- ion they express on a given subject (Pang et al., 2002). Gen- erally speaking, opinion mining aims at determining the at- titude of a speaker or a writer with respect to a topic or the overall tonality of a document. This task has created a considerable interest due to its wide applications. In recent years, the exponential increase of the Web for exchanging public opinions about events, facts, products, etc., led to an extensive usage of opinion mining approaches, especially for marketing purposes. By formalizing the opinion mining problem, an “opinion” (or “sentiment”) has been defined by Liu and Zhang (2012) as a quintuple: ⟨oj, fjk, soijkl, hi, tl⟩, (1) where oj is a target object, fjk is a feature of the object oj, soijkl is the sentiment value held by person hi (which will be called an “opinion holder”) on feature fjk of object oj at time tl. The value of soijkl can be positive (denoting a state of happiness, bliss, or satisfaction), negative (denoting a state of sorrow, dejection, or disappointment), or neutral (it is not possible to denote any particular sentiment), or a more granular rating. The term hi encodes the opinion holder, and tl is the time when the opinion is expressed. With the growth of the number of solutions available in this field, the comparison between new approaches and the ones available in the state of the art is a mandatory task for measuring the quality of proposed solution. Unfortunately, in many cases it is not trivial to replicate evaluation cam- paigns mentioned into published papers. Among all possi- ble issues, like the difficulty in replicating the implemented model, we want to focus on the “data” side. More in detail, the issues that can be easily found when we try to replicate experiments are the following. • The used dataset is not longer available for download. • The system is not longer maintained or available on the authors’ website. • In case of dataset availability, it is not clear how the dataset has been split for training …

Content truncated to 500 words