ERNIE at SemEval-2020 Task 10: Learning Word Emphasis Selection by Pre-trained Language Model

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1456–1461 Barcelona, Spain (Online), December 12, 2020. 1456 ERNIE at SemEval-2020 Task 10: Learning Word Emphasis Selection by Pre-trained Language Model Zhengjie Huang, Shikun Feng, Weiyue Su, Xuyi Chen, Shuohuan Wang, Jiaxiang Liu, Xuan Ouyang, Yu Sun Baidu Inc., China {huangzhengjie,fengshikun01,suweiyue,chenxuyi}@baidu.com {wangshuohuan,liujiaxiang,ouyangxuan,sunyu02}@baidu.com Abstract This paper describes the system designed by ERNIE Team which achieved the first place in SemEval-2020 Task 10: Emphasis Selection For Written Text in Visual Media. Given a sentence, we are asked to find out the most important words as the suggestion for automated design. We leverage the unsupervised pre-training model and finetune these models on our task. After our investigation, we found that the following models achieved an excellent performance in this task: ERNIE 2.0, XLM-ROBERTA, ROBERTA and ALBERT. We combine a pointwise regression loss and a pairwise ranking loss which is more close to the final Matchm metric to finetune our models. And we also find that additional feature engineering and data augmentation can help improve the performance. Our best model achieves the highest score of 0.823 and ranks first for all kinds of metrics. 1 Introduction Emphasis selection for written text in visual media is proposed by Shirani et al. (2020) and Shirani et al. (2019). The purpose of this shared task is to design automatic methods for emphasis selection, i.e. choosing candidates for emphasis in short written text, to enable automated design assistance in authoring. For example, Shirani et al. (2019) mentions that such a technique can be applied to some graphic design applications such as Adobe Spark to perform automatic text layout using templates that include images and text with different fonts and colors. The major challenge is that given only thousands of annotated short text data without any context about the text or visual background images, we are asked to learn the author- or domain-specific emphatic about the short text. Besides, these short text data are annotated by crowd-sourcing workers. And we find that different annotators have different standards, which increases the difficulty of this task. To identify the most important words, we model the task as a sequential labeling problem. Our base models leverage different unsupervised language model such as ERNIE 2.0 (Sun et al., 2019b), XLM- ROBERTA (Conneau et al., 2019), ROBERTA (Liu et al., 2019) and ALBERT (Lan et al., 2019). These large unsupervised models are pre-trained on a large amount of unannotated data and carry valuable lexical, syntactic, and semantic information in training corpora. Our approach is as follows: firstly, the word-level output representations for the sentence are computed by pre-trained models and then fed into a designed downstream neural network for word selections; secondly, we finetune the downstream networks together with the pre-trained models on the annotated training data; thirdly, we investigate several different objective functions to learn our model; and finally, we apply feature engineering and several data augmentation strategies for further improvement. The rest of the paper is organized as follows. In Section 2, …

Content truncated to 500 words