Hitachi at SemEval-2020 Task 10: Emphasis Distribution Fusion on Fine-Tuned Language Models

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1658–1664 Barcelona, Spain (Online), December 12, 2020. 1658 Hitachi at SemEval-2020 Task 10: Emphasis Distribution Fusion on Fine-Tuned Language Models Gaku Morio∗, Terufumi Morishita*, Hiroaki Ozaki and Toshinori Miyoshi Hitachi, Ltd. Resarch and Development Group Kokubunji, Tokyo, Japan {gaku.morio.vn, terufumi.morishita.wp, hiroaki.ozaki.yu, toshinori.miyoshi.pd}@hitachi.com Abstract This paper shows our system for SemEval-2020 task 10, Emphasis Selection for Written Text in Visual Media. Our strategy is two-fold. First, we propose fine-tuning many pre-trained language models, predicting an emphasis probability distribution over tokens. Then, we propose stacking a trainable distribution fusion (DISTFUSE) system to fuse the predictions of the fine-tuned models. Experimental results show that DISTFUSE is comparable or better when compared with a naive average ensemble. As a result, we were ranked 2nd amongst 31 teams. 1 Introduction This paper presents our strategy for SemEval-2020 task 10, Emphasis Selection for Written Text in Visual Media (Shirani et al., 2020). The task is aimed at emphasis selection, choosing candidates for emphasis in short written text in visual media (Shirani et al., 2019). Rather than predicting emphasis spans or using images, the task involves the prediction of an emphasis distribution over a short text without any image inputs. BERT T5 GPT-2 RoBERTa Fine-tune and hyperparameter search with cross validation ・・・ Nothing isso contagiou sas example . Input: Nothing is so contagious as example . ・・・ DistFuse ・・・ Figure 1: Overview of our strategy We tackle the task by combining rich contextualized embed- dings of many fine-tuned pre-trained language models (PLMs). Our strategy, shown in Figure 1, is a simple but effective meta ensemble method. First, we fine-tune heterogeneous PLMs such as BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), predicting an emphasis distribution over tokens. The models are trained with the KL-divergence of gold-emphasis distributions over tokens in a text fragment to handle the am- biguity within annotations (Shirani et al., 2019). Second, we propose distribution fusion (DISTFUSE) for ensembles. Dif- ferent from an averaging ensemble, DISTFUSE assigns a kind of reliability weight to each distribution. Hence, accuracy can be improved with the method. We evaluate the proposed system in large-scale experiments that suggest that DISTFUSE is comparable or better when com- pared with the average ensemble. As a result, our system ranked 2nd amongst 31 teams. We also provide interesting insights such as on the distinct performance of PLMs, training techniques, and hyperparame- ters in the results section. 2 Background The modeling of word emphasis has been widely tackled in some contexts. Zhang et al. (2016) proposed a model for extracting key phrases from Twitter text. In the context of accent, Nakajima et al. (2014) ∗Contributed equally. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 1659 val1 fold2 fold3 fold1 fold3 fold1 fold2 val1 val2 val3 model (fold1) model (fold2) model (fold3) val2 val3 predict train folds ・・・ valid fold DistFuse predict & train Train with hyperparameter set 2 PLM = BERT ・・・ …

Content truncated to 500 words