IITK at SemEval-2020 Task 10: Transformers for Emphasis Selection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1665–1670 Barcelona, Spain (Online), December 12, 2020. 1665 IITK at SemEval-2020 Task 10: Transformers for Emphasis Selection Vipul Singhal∗ Sahil Dhull∗ Rishabh Agarwal∗ Indian Institute of Technology Kanpur (IITK)Ashutosh Modi {vipulsn,sahild,rish}@iitk.ac.in ashutoshm@cse.iitk.ac.in Abstract This paper describes the system proposed for addressing the research problem posed in Task 10 of SemEval-20201: Emphasis Selection For Written Text in Visual Media. We propose an end-to-end model that takes as input the text and corresponding to each word gives the probability of the word to be emphasized. Our results show that transformer-based models are particularly effective in this task. We achieved the best Matchm score (described in section 2.2) of 0.810 and were ranked third on the leader-board. 1 Introduction Visual communication relies on images and short texts, e.g., in flyers, posters, etc. The purpose of these is to convey the message effectively and without ambiguity. Moreover, they should be able to attract the reader’s attention within the first few seconds. For text, this can be achieved by laying emphasis on particular words to convey the intent better. The emphasis selection task is about designing automatic methods for choosing candidate words to be emphasized in short written texts, to enable automated design assistance in authoring (Figure 1). The dataset provided for this task is in English, and task description paper (Shirani et al., 2020) provided by the organizers describes the task, data, evaluation, results, and a summary of participating systems. We tried two different approaches for the task. Our BiLSTM + Attention approach is inspired by the baseline paper (Shirani et al., 2019). In this approach, we tweaked the BiLSTM layers (Hochreiter and Schmidhuber, 1997), tried other layers like GRU (Cho et al., 2014), and used character embeddings along with word embeddings (Lample et al., 2016). Our Transformers approach involves transfer learning using Transformer based models (Vaswani et al., 2017). This approach involves two types of models, the first one being a transformer-based model with the BiLSTM layer, the attention layer (Bahdanau et al., 2014), and fully connected layers on top. The second type of model involves transformer-based models with fully connected layers. We used BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019), and GPT-2 (Radford et al., 2019) as transformer-based models. In the end, we tried the homogeneous and heterogeneous ensemble of these models. Due to the nature of the problem, the systems which incorporate the whole context of a sentence would perform better. Further, the small size of the dataset was a bottleneck that can be countered by using transfer learning via the pre-trained models. Using transformer-based models accounts for both of these observations. Our best submission achieved Matchm score of 0.810 (Matchm evaluation metric is described in Section 2.2) and was ranked third on the leaderboard. Our submissions under-performed in Score 1 (Match1 as defined in Section 2.2) as compared to the other top-performing teams. Our code is available on Github2. This work is …

Content truncated to 500 words