Comparing Approaches to Language Understanding for Human-Robot Dialogue: An Error Taxonomy and Analysis

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 581–589 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 581 Multi-source Multi-domain Sentiment Analysis with BERT-based Models Gabriel Roccabruna, Steve Azzolin, Giuseppe Riccardi Signals and Interactive Systems Lab, DISI, University of Trento gabriel.roccabruna@unitn.it, steve.azzolin@studenti.unitn.it, giuseppe.riccardi@unitn.it Abstract Sentiment analysis is one of the most widely studied tasks in natural language processing. While BERT-based models have achieved state-of-the-art results in this task, little attention has been given to its performance variability across class labels, multi-source and multi-domain corpora. In this paper, we present an improved state-of-the-art and comparatively evaluate BERT-based models for sentiment analysis on Italian corpora. The proposed model is evaluated over eight sentiment analysis corpora from different domains (social media, finance, e-commerce, health, travel) and sources (Twitter, YouTube, Facebook, Amazon, Tripadvisor, Opera and Personal Healthcare Agent) on the prediction of positive, negative and neutral classes. Our findings suggest that BERT-based models are confident in predicting positive and negative examples but not as much with neutral examples. We release the sentiment analysis model as well as a newly financial domain sentiment corpus. Keywords: sentiment analysis, multi-domain, multi-source 1. Introduction By now most companies including social media, e- commerce, forecasting companies are collecting and an- alyzing customers’ opinions. These opinions can be ex- tracted from big data produced every day on the Internet using sentiment analysis. Sentiment analysis is the natu- ral language processing task that automatically extracts the writer’s orientation from written text (Cardie, 2014). Senti- ment analysis has been effectively used to extract attitudes from movie reviews (Kumar et al., 2019), Amazon reviews (Ejaz et al., 2017), and Twitter (Rosenthal et al., 2017). A widely used technique for sentiment analysis is based on lexicons (Khoo and Johnkhan, 2018; Taboada et al., 2011). Lexicon-based methods assign a score, ranging from posi- tive to negative, to each word in the text. These scores are aggregated into a global score which represents the senti- ment polarity associated to a text segment. The most used polarity label dictionary is ternary: positive, negative, or neutral. The assumption behind lexicon-based methods is that the sentiment polarity can be inferred from the fre- quency and the strength of sentiment-charged words, such as horrible, good or terrific. However, a draw- back of lexicon-based methods is that they consider the text as a bag of words, limiting the interpretation of natural lan- guage in context. The semantic content of a text is a cru- cial aspect for detecting sentiments that are not anchored to a single word or word chunk. This issue is partially addressed by BERT-based (Kenton and Toutanova, 2019) models, which compute the semantic content representa- tion of a text by exploiting the whole word context. In recent years, BERT-based models have achieved state- of-the-art results in the sentiment analysis task (Polignano et al., 2019; Xie et al., 2020; Bianchi et al., 2021). How- ever, little research has been conducted on the perfor- mance variability of BERT-based models on …

Content truncated to 500 words