Tasty Burgers, Soggy Fries: Probing Aspect Robustness in Aspect-Based Sentiment Analysis

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3594–3605, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3594 Tasty Burgers, Soggy Fries: Probing Aspect Robustness in Aspect-Based Sentiment Analysis Xiaoyu Xing1∗ Zhijing Jin2∗ Di Jin3 Bingning Wang4 Qi Zhang1 Xuanjing Huang1 1Fudan University 2Max Planck Institute, 3CSAIL, MIT, 4Sogou Inc. {xyxing18,qz,xjhuang}@fudan.edu.cn zjin@tuebingen.mpg.de jindi15@mit.edu, wangbingning@sogou-inc.com Abstract Aspect-based sentiment analysis (ABSA) aims to predict the sentiment towards a specific as- pect in the text. However, existing ABSA test sets cannot be used to probe whether a model can distinguish the sentiment of the target as- pect from the non-target aspects. To solve this problem, we develop a simple but effective ap- proach to enrich ABSA test sets. Specifically, we generate new examples to disentangle the confounding sentiments of the non-target as- pects from the target aspect’s sentiment. Based on the SemEval 2014 dataset, we construct the Aspect Robustness Test Set (ARTS) as a com- prehensive probe of the aspect robustness of ABSA models. Over 92% data of ARTS show high fluency and desired sentiment on all as- pects by human evaluation. Using ARTS, we analyze the robustness of nine ABSA models, and observe, surprisingly, that their accuracy drops by up to 69.73%. We explore several ways to improve aspect robustness, and find that adversarial training can improve models’ performance on ARTS by up to 32.85%.1 1 Introduction Aspect-based sentiment analysis (ABSA) is an ad- vanced sentiment analysis task that aims to classify the sentiment towards a specific aspect (e.g., burg- ers or fries in the review “Tasty burgers, and crispy fries.”). The key to a strong ABSA model is it is sensitive to only the sentiment words of the tar- get aspect, and therefore not be interfered by the sentiment of any non-target aspect. Although state- of-the-art models have shown high accuracy on existing test sets, we still question their robustness. Specifically, given the prerequisite that a model out- puts correct sentiment polarity for the test example, we have the following questions: ∗ Equal Contribution. Please email (or cc’) both first authors for correspondence. 1Our code and new test set are available at https:// github.com/zhijing-jin/ARTS_TestSet. (Q1) If we reverse the sentiment polarity of the tar- get aspect, can the model change its prediction accordingly? (Q2) If the sentiments of all non-target aspects be- come opposite to the target one, can the model still make the correct prediction? (Q3) If we add more non-target aspects with sen- timents opposite to the target one, can the model still make the correct prediction? A robust ABSA model should both meet the prerequisite and have affirmative answers to all the questions above. For example, if a model makes the correct sentiment classification (i.e., positive) for burgers in the original sentence “Tasty burgers, and crispy fries”, it should flip its prediction (to negative) when seeing the new context “Terrible burgers, but crispy fries”. Hence, these questions together form a probe to verify if an ABSA model has high aspect robustness. Unfortunately, existing …

Content truncated to 500 words