FPAI at SemEval-2020 Task 10: A Query Enhanced Model with RoBERTa for Emphasis Selection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1652–1657 Barcelona, Spain (Online), December 12, 2020. 1652 FPAI at SemEval-2020 Task 10: A Query Enhanced Model with RoBERTa for Emphasis Selection Chenyang Guo*, Xiaolong Hou* , Junsong Ren, Lianxin Jiang, Yang Mo, Haiqin Yang, Jianping Shen Ping An Life Insurance, Lt ShenZhen, China {guochenyang210, houxiaolong430, renjunsong941, jianglianxin769, moyang853, yanghaiqin260, shenjianping324}@pingan.com.cn Abstract This paper describes the model we apply in the SemEval-2020 Task 10. We formalize the task of emphasis selection as a simplified query-based machine reading comprehension (MRC) task, i.e. answering a fixed question of “Find candidates for emphasis”. We propose our subword puzzle encoding mechanism and subword fusion layer to align and fuse subwords. By introducing the semantic prior knowledge of the informative query and some other techniques, we attain the 7th place during the evaluation phase and the first place during train phase. 1 Introduction The SemEval-2020 Task 10 focuses on the emphasis selection in visual text media, i.e. choosing candidates for emphasis in short written text, to enable automated design assistance in authoring (Shirani et al., 2019). In this paper, we propose a novel model enhanced by an informative query with RoBERTa (Liu et al., 2019) as the backbone to select the emphasis in a sentence. We are inspired by Li et al. (2019) who introduced a MRC approach to improve the performance of the NER task by querying differently for diverse named entities. We reform the MRC query and formalize the emphasis selection task as a simplified question answering task in machine reading comprehension (MRC). That is, given a natural language query, emphasis tokens are extracted to answer the query. For example, the task of selecting emphasis tokens “dangerous”, “evolve”, “not”, “to” in “What ’s dangerous is not to evolve” is formalized as an MRC task of answering a fixed query, “Find candidates for emphasis”. Additionally, we propose subword puzzle encoding and the subword fusion layer to solve the token alignment problem gained from RoBERTa. Our work shows that the informative query is beneficial in the case of emphasis selection. 2 Background The purpose of this task is to design automatic methods for selecting emphasis on visual media such as flyers, posters and ads. The selected emphasis, written in English, is highlighted to grab a viewer’s attention. The competition dataset consists of 3,143 sentences with labels and 743 sentences without labels. Details about the dataset are introduced in (Shirani et al., 2019) and (Shirani et al., 2020). Most of sentences in the dataset contain a few words, usually less than 20. 3 Model Figure 1 depicts our proposed model: given a sentence, we apply concatenation to the uniform query and the sentence to obtain the input. Then we apply subword puzzle encoding to store the affiliation information of subwords. After that, we apply RoBERTa to get the representation of each token, i.e. the embeddings from the last two hidden layers in RoBERTa. They are then fused in the subword fusion layer and output by a …

Content truncated to 500 words