UIC-NLP at SemEval-2020 Task 10: Exploring an Alternate Perspective on Evaluation

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1704–1709 Barcelona, Spain (Online), December 12, 2020. 1704 UIC-NLP at SemEval-2020 Task 10: Exploring an Alternate Perspective on Evaluation Philip Hossu University of Illinois at Chicago Department of Mathematics, Statistics, and Computer Science phossu2@uic.edu Natalie Parde University of Illinois at Chicago Department of Computer Science parde@uic.edu Abstract In this work we describe and analyze a supervised learning system for word emphasis selection in phrases drawn from visual media as a part of the Semeval 2020 Shared Task 10. More specifically, we begin by briefly introducing the shared task problem and provide an analysis of interesting and relevant features present in the training dataset. We then introduce our LSTM-based model and describe its structure, input features, and limitations. Our model ultimately failed to beat the benchmark score, achieving an average match() score of 0.704 on the validation data (0.659 on the test data) but predicted 84.8% of words correctly considering a 0.5 threshold. We conclude with a thorough analysis and discussion of erroneous predictions with many examples and visualizations. 1 Introduction and Task Background Short phrases are a frequent form of communication, particularly in visual media (e.g., advertisements and motivational quotes). It is common to see one or more of these words emphasized, whether it be through bolding, italicizing, or changing font size or color. Carefully selecting word emphasis can be key for effective advertising and clear messaging; thus, automating this task offers intriguing real-world potential. In this shared task, researchers sought to develop systems which accurately emphasize words in variable length text phrases, drawn originally from various forms of visual media. While there have been similar past investigations into domain-specific emphasis selection models, this task was based on a diverse set of phrases, ranging from inspirational quotes (e.g., Only in the darkness can you see the stars) to advertisements (e.g., Bastille Day Block Party) and beyond. In all cases these phrases originated from some form of visual media, such as online graphics or printed posters/advertisements. The task is highly subjective—for the aforementioned “block party,” one could argue that any number of emphasis options are appropriate, from “Bastille Day Block Party,” to “Bastille Day Block Party,” or maybe even “Bastille Day Block Party.” This challenging, inherent subjectivity limits the performance of any model created for the task. This task is further described by Shirani et al. (2020). 2 Data Analysis This task made use of a dataset obtained from Adobe Spark (https://spark.adobe.com). The provided training set contained 2742 unique phrases, comprising a total of 32400 words. These phrases were annotated manually by nine individuals who decided, for each word, whether the word should be emphasized. The individual labels were aggregated by the task organizers, with the final gold standard being provided as ratios of votes for emphasized over votes for not emphasized, ranging from {0.0, 0.11, ..., 1.0}. A similarly-labeled validation set containing 392 phrases (a total of 4385 words) was also provided. Before creating models, we analyzed the data to …

Content truncated to 500 words