Opinion Prediction with User Fingerprinting
20182Authors
nan
Abstract
AbstractOpinion prediction is an emerging research area with diverse real-world applications, such as market research and situational awareness. We identify two lines of approaches to the problem of opinion prediction. One uses topic-based sentiment analysis with time-series modeling, while the other uses static embedding of text. The latter approaches seek user-specific solutions by generating user fingerprints. Such approaches are useful in predicting user’s reactions to unseen content. In this work, we propose a novel dynamic fingerprinting method that leverages contextual embedding of user’s comments conditioned on relevant user’s reading history. We integrate BERT variants with a recurrent neural network to generate predictions. The results show up to 13% improvement in micro F1-score compared to previous approaches. Experimental results show novel insights that were previously unknown such as better predictions for an increase in dynamic history length, the impact of the nature of the article on performance, thereby laying the foundation for further research.
Keywords
Full Content
Proceedings of Recent Advances in Natural Language Processing, pages 121–127 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_015 121 Litescale: A Lightweight Tool for Best-worst Scaling Annotation Valerio Basile University of Turin valerio.basile@unito.it Christian Cagnazzo University of Turin christian.cagnazzo@edu.unito.it Abstract Best-worst Scaling (BWS) is a methodol- ogy for annotation based on comparing and ranking instances, rather than classifying or scoring individual instances. Studies have shown the efficacy of this methodology ap- plied to NLP tasks in terms of a higher qual- ity of the datasets produced by following it. In this system demonstration paper, we present LITESCALE, a free software library to create and manage BWS annotation tasks. LITESCALE computes the tuples to annotate, manages the users and the annotation process, and creates the final gold standard. The func- tionalities of LITESCALE can be accessed pro- grammatically through a Python module, or via two alternative user interfaces, a textual console-based one and a graphical Web-based one. We further developed and deployed a fully online version of LITESCALE complete with multi-user support. 1 Introduction Annotation is a cornerstone of Computational Lin- guistics and Natural Language Processing (NLP). Much of modern NLP is based on machine learning, and in particular on supervised machine learning. As a consequence, large amounts of manually an- notated data are constantly in high demand, and the quality of the annotation directly influence the predictive capability of models trained on them. The annotation of natural language resources comes in many forms of varying complexity and levels of abstraction. However, it is possible to categorize most of the approaches in three main families. Categorial annotation is perhaps the most common approach, whereas each instance of a dataset is associated with a label from a fixed set of options. Annotation can also consist of scalar values, that is, numeric values on a predetermined scale. Finally, ranking is a type of annotation where multiple instances are put in a certain order by the annotator, who therefore does not make judgments on instances in isolation but rather on groups of them. Recent literature highlights the advantages of the ranking strategy, in terms of the quality of the anno- tation produced with it, in particular when dealing with subjective aspects of natural language (Yan- nakakis et al., 2018). The Best-Worst Scaling model (BWS) is a ranking-based annotation pro- cess developed by Louviere et al. (2015). BWS has been proved to be beneficial to the quality of the resulting gold standard data for subjective-related tasks such as emotion detection (Kiritchenko and Mohammad, 2017) and hate speech detection (Po- letto et al., 2019). The recent work of De Bruyne, Luna and De Clercq, Orph´ee and Hoste, Veronique (2021) further proved the validity of BWS as a method of annotation for sentiment-related tasks, dominance in particular. Several general-purpose annotation tools have been proposed in the literature. Among the most popular, WebAnno (Eckart de Castilho et al., 2016) and Brat (Stenetorp et al., 2012) both provide Web- based interfaces and the possibility to implement rich annotation schemas, including annotation at the …
Content truncated to 500 words
Labeled Text
AbstractOpinion prediction is an emerging research area with diverse real-world applications, such as market research and situational awareness. We identify two lines of approaches to the problem of opinion prediction. One uses topic-based sentiment analysis with time-series modeling, while the other uses static embedding of text. The latter approaches seek user-specific solutions by generating user fingerprints. Such approaches are useful in predicting user’s reactions to unseen content. In this work, we propose a novel dynamic fingerprinting method that leverages contextual embedding of user’s comments conditioned on relevant user’s reading history. We integrate BERT variants with a recurrent neural network to generate predictions. The results show up to 13% improvement in micro F1-score compared to previous approaches. Experimental results show novel insights that were previously unknown such as better predictions for an increase in dynamic history length, the impact of the nature of the article on performance, thereby laying the foundation for further research.
Publication Details
- Paper ID:
- 20182
- Year:
- 2021
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan