YASO: A Targeted Sentiment Analysis Evaluation Dataset for Open-Domain Reviews

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9154–9173 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9154 YASO: A Targeted Sentiment Analysis Evaluation Dataset for Open-Domain Reviews Matan Orbach, Orith Toledo-Ronen, Artem Spector, Ranit Aharonov, Yoav Katz, Noam Slonim IBM Research {matano, oritht, artems, katz, noams}@il.ibm.com ranit.aharonov2@ibm.com Abstract Current TSA evaluation in a cross-domain setup is restricted to the small set of review domains available in existing datasets. Such an evaluation is limited, and may not reflect true performance on sites like Amazon or Yelp that host diverse reviews from many domains. To address this gap, we present YASO – a new TSA evaluation dataset of open-domain user reviews. YASO contains 2215 English sentences from dozens of review domains, an- notated with target terms and their sentiment. Our analysis verifies the reliability of these an- notations, and explores the characteristics of the collected data. Benchmark results using five contemporary TSA systems show there is ample room for improvement on this chal- lenging new dataset. YASO is available at github.com/IBM/yaso-tsa. 1 Introduction Targeted Sentiment Analysis (TSA) is the task of identifying the sentiment expressed towards sin- gle words or phrases in texts. For example, given the sentence "it’s a useful dataset with a complex download procedure" the desired output is iden- tifying dataset and download procedure, with a positive and negative sentiments expressed towards them, respectively. Our focus in this work is on TSA of user reviews data in English. Till recently, typical TSA evaluation was in- domain, for example, by training on labeled restau- rant reviews and testing on restaurant reviews. New works (e.g. Rietzler et al. (2020)) began consider- ing a cross-domain setup, training models on la- beled data from one or more domains (e.g., restau- rant reviews) and evaluating on others (e.g., lap- top reviews). For many domains, such as car or book reviews, TSA data is scarce or non-existent. This suggests that cross-domain experimentation is more realistic, as it aims at training on a small set of labeled domains and producing predictions for reviews from any domain. Naturally, the evaluation in this setup should resemble real-world content from sites like Amazon or Yelp that host reviews from dozens or even hundreds of domains.1 Existing English TSA datasets do not facilitate such a broad evaluation, as they typically include reviews from a small number of domains. For ex- ample, the popular SEMEVAL (SE) datasets created by Pontiki et al. (2014, 2015, 2016) (henceforth SE14, SE15, and SE16, respectively), contain En- glish reviews of restaurants, laptops and hotels (see §2 for a discussion of other existing datasets). To address this gap, we present YASO,2 a new TSA dataset collected over user reviews taken from four sources: the YELP and AMAZON (Keung et al., 2020) datasets of reviews from those two sites; the Stanford Sentiment Treebank (SST) movie reviews corpus (Socher et al., 2013); and the OPINOSIS dataset of reviews from over 50 topics (Ganesan et al., 2010). To the best of our knowledge, …

Content truncated to 500 words