TëXtmarkers at SemEval-2020 Task 10: Emphasis Selection with Agreement Dependent Crowd Layers

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1698–1703 Barcelona, Spain (Online), December 12, 2020. 1698 T¨extmarkers at SemEval-2020 Task 10: Emphasis Selection with Agreement Dependent Crowd Layers Kevin Glocker and Stefanos Andreas Markianos Wright Department of Linguistics University of T¨ubingen {kevin.glocker,stefanos-andreas.markianos-wright}@student.uni-tuebingen.de Abstract In visual communication, the ability of a short piece of text to catch someone’s eye in a single glance or from a distance is of paramount importance. In our approach to the SemEval-2020 task “Emphasis Selection For Written Text in Visual Media”, we use contextualized word rep- resentations from a pretrained model of the state-of-the-art BERT architecture together with a stacked bidirectional GRU network to predict token-level emphasis probabilities. For tackling low inter-annotator agreement in the dataset, we attempt to model multiple annotators jointly by introducing initialization with agreement dependent noise to a crowd layer architecture. We found our approach to both perform substantially better than initialization with identities for this purpose and to outperform a baseline trained with token level majority voting. Our submission system reaches substantially higher Matchm on the development set than the task baseline (0.779), but only slightly outperforms the test set baseline (0.754) using a three model ensemble. 1 Introduction Emphasis selection is the task of choosing individual words or phrases of short written texts to emphasize. While emphasis selection has also been used in previous work for more natural articulation in text-to- speech systems (Mass et al., 2018), in the SemEval shared task emphasis annotations are intended for use in visual communication such as in posters or advertisements (Shirani et al., 2020). Word emphasis helps readers conveniently scan for important information in a text. Capital letters, bold letters, italics and underlines can help emphasize certain facts and drive the message home; however, overuse of these features can lead to confusion and create an unattractive aesthetic. Usually advertisers focus on one or two key words or phrases that will pique the readers’ interest (Shirani et al., 2019). As with many tasks based on sentence modeling, substantial improvements have been gained in emphasis selection by leveraging recent breakthroughs in language modeling such as the ELMo architec- ture (Peters et al., 2018; Shirani et al., 2019). Following this trend, our submission uses contextualized word representations from a pretrained model of the state-of-the-art BERT architecture (Devlin et al., 2019) together with a stacked bidirectional GRU network for learning task specific information. A major challenge when developing a system for automatic emphasis selection are very low agreements between annotators about which parts of a sentence should be emphasized which impacts how models are trained and evaluated (Mass et al., 2018; Shirani et al., 2019). To tackle annotator subjectivity, both majority voting (Mass et al., 2018) and regression with a KL divergence objective (Shirani et al., 2019) have been used for training in previous work. In our approach, we apply a crowd layer architecture adapted from Rodrigues and Pereira (2018) and improve its effectiveness for the task by introducing agreement dependent noise during initialization. 2 …

Content truncated to 500 words