IDS at SemEval-2020 Task 10: Does Pre-trained Language Model Know What to Emphasize?

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1371–1376 Barcelona, Spain (Online), December 12, 2020. 1371 IDS at SemEval-2020 Task 10: Does Pre-trained Language Model Know What to Emphasize? Jaeyoul Shin, Taeuk Kim and Sang-goo Lee Dept. of Computer Science and Engineering Seoul National University, Seoul, Korea {jyshin,taeuk,sglee}@europa.snu.ac.kr Abstract We propose a novel method that enables us to determine words that deserve to be emphasized from written text in visual media, relying only on the information from the self-attention distributions of pre-trained language models (PLMs). With extensive experiments and analyses, we show that 1) our zero-shot approach is superior to a reasonable baseline that adopts TF-IDF and that 2) there exist several attention heads in PLMs specialized for emphasis selection, confirming that PLMs are capable of recognizing important words in sentences. 1 Introduction In terms of visual communication such as social media posts (posters, flyers, and ads) and motivational messages, text emphasis is crucial in that it facilitates the comprehension of written text and that it helps convey the author’s intent. Therefore, it is expected that the development of an automatic system that recommends which part to emphasize in the text of visual media will bring significant advantages; e.g., it is possible to accelerate the making process of posters and videos for advertisement. In this paper, we attempt to devise such a system solely with the aid of pre-trained language models (PLMs) without any task-specific laborious training. Recently, there has been a substantial amount of work in the literature to figure out what knowledge Transformer (Vaswani et al., 2017) based PLMs such as BERT (Devlin et al., 2019) contain and why they perform surprisingly well on various downstream tasks (Goldberg, 2019; Kovaleva et al., 2019; Rogers et al., 2020). Among these, a group of studies has focused on analyzing PLMs’ self-attention distributions to find some evidence that supports the existence of linguistic knowledge within the pre-trained weights of PLMs. Specifically, Clark et al. (2019) investigated BERT’s individual attention heads to probe its ability to parse dependency trees, while Kim et al. (2020) proposed an unsupervised constituency parsing method applicable on top of the distributions. Following the same philosophy shared by the work mentioned above, we propose a zero-shot emphasis selection method, assuming that during pre-training, some attention heads in PLMs are inspired to recognize which words are more important than others. We test our method on a carefully designed dataset (Shirani et al., 2020), and it records the ranking score 0.690 on the validation set, which outperforms an intuitive baseline adopting the term frequency-inverse document frequency (TF-IDF) strategy (Jones, 1972). Furthermore, our method operates in a fully zero-shot manner, not leveraging gold-standard annotations provided by the dataset at all, implying its universal applicability in a low/zero-resource regime. 2 Task This paper aims to present a tractable solution to the SemEval 2020 shared task 10 (Shirani et al., 2020). The dataset provided by this task consists of short English sentences obtained from Adobe Spark. It contains a variety of …

Content truncated to 500 words