Why Do Document-Level Polarity Classifiers Fail?

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1782–1794 June 6–11, 2021. ©2021 Association for Computational Linguistics 1782 Why Do Document-Level Polarity Classifiers Fail? Karen S. Martins CS Department UFMG Belo Horizonte, MG, Brazil karensm@dcc.ufmg.br Pedro O.S. Vaz-de-Melo CS Department UFMG Belo Horizonte, MG, Brazil olmo@dcc.ufmg.br Rodrygo L. T. Santos CS Department UFMG Belo Horizonte, MG, Brazil rodrygo@dcc.ufmg.br Abstract Machine learning solutions are often criticized for the lack of explanation of their successes and failures. Understanding which instances are misclassified and why is essential to im- prove the learning process. This work helps to fill this gap by proposing a methodology to characterize, quantify and measure the impact of hard instances in the task of polarity clas- sification of movie reviews. We characterize such instances into two categories: neutrality, where the text does not convey a clear polarity, and discrepancy, where the polarity of the text is the opposite of its true rating. We quantify the number of hard instances in polarity classi- fication of movie reviews and provide empiri- cal evidence about the need to pay attention to such problematic instances, as they are much harder to classify, for both machine and human classifiers. To the best of our knowledge, this is the first systematic analysis of the impact of hard instances in polarity detection from well- formed textual reviews. 1 Introduction Document-level polarity classification is the task of classifying the polarity of a whole opinionated message (Pozzi et al., 2016). For instance, given a movie review, the system determines whether the review text expresses an overall positive, neg- ative, or neutral opinion about the movie. Al- though polarity classification naturally suits to an- alyze consumer opinions about products and ser- vices (Gui et al., 2017), it is also well suited to various types of applications, such as to infer votes in elections (Goldberg et al., 2007), civilian senti- ment during terrorism scenarios (Cheong and Lee, 2011), citizens’ perception of government agen- cies (Arunachalam and Sarkar, 2013) and recom- mendation systems (Zhang, 2015). Supervised machine learning is one of the most common and successful approaches for polarity classification, but even state-of-the-art methods fail to correctly classify a substantial portion of the instances, from 10% to 20%, depending on the dataset (Ribeiro et al., 2016). The problem with this approach is that if the data is not representative and reliable, the model is unlikely to perform well. One source of unreliability is data noise, which can be categorized into class noise and attribute noise (Gupta and Gupta, 2019). Class noise occurs when the training data contains instances that are wrongly labeled. Attribute noise occurs when the training data contains one or more attributes with wrong, incomplete or missing values. In the case of textual data, such noise usually comes in the form of errors in language rules, such as typos, gram- matical errors, improper punctuation, and abbrevi- ations (Agarwal et al., 2007; Michel and Neubig, 2018; Lourentzou et …

Content truncated to 500 words