LAST at SemEval-2020 Task 10: Finding Tokens to Emphasise in Short Written Texts with Precomputed Embedding Models and LightGBM

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1671–1677 Barcelona, Spain (Online), December 12, 2020. 1671 LAST at SemEval-2020 Task 10: Finding tokens to emphasise in short written texts with precomputed embedding models and LightGBM Yves Bestgen Laboratoire d’analyse statistique des textes - LAST Institut de recherche en sciences psychologiques Universit´e catholique de Louvain Place Cardinal Mercier, 10 1348 Louvain-la-Neuve, Belgium yves.bestgen@uclouvain.be Abstract To select tokens to be emphasised in short texts, a system mainly based on precomputed embedding models, such as BERT and ELMo, and LightGBM is proposed. Its performance is low. Additional analyzes suggest that its effectiveness is poor at predicting the highest emphasis scores while they are the most important for the challenge and that it is very sensitive to the specific instances provided during learning. 1 Introduction This paper reports on the participation of the Laboratoire d’analyse statistique des textes (LAST) in the SemEval-2020 Task 10 Emphasis Selection For Written Text in Visual Media. This task requires automatic systems to select tokens1 for emphasis in English famous quotes (e.g., No matter how hard you work, someone else is working harder.2) and other short written text such the ones used in flyers, advertisements or social media posts (e.g., stay positive). Its importance is growing continuously given the explosion in the number of slogans that web users want to display on smaller and smaller screens of more and more devices. Writing brief, attention-grabbing and engaging message has become paramount. One of the characteristics of this task which makes it particularly complex resides in the brevity of the texts which means that almost all words, with the notable exception of grammatical words, are important otherwise they would have been omitted. So, it is more about identifying the most important words on a scale than making a binary decision between important vs. not important. This task has not attracted attention until last year when Shirani et al. (2019) simultaneously highlighted its interest, collected from Adobe Spark3 a dataset annotated by human judges, and proposed an automatic system for performing it. This system uses a deep sequence labeling network composed of two stacked bidirectional LSTM layers and relies on ELMo embeddings. Shirani et al. (2019) observed that it outperformed a Conditional Random Fields model (Lafferty et al., 2001) with hand-crafted features. However, they did not evaluate other classical supervised learning techniques, such as gradient boosting decision tree (Friedman, 2000), or the combination of these techniques with precomputed embedding models. The main objective of this study is to try to determine whether the use of such procedures can be competitive for the emphasis selection task. The remainder of this report describes the datasets made available for this challenge and the evaluation criteria, the systems developed, and the results obtained as well as a few analyses performed to get a better idea of the factors that affect the system performance. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 1The term token is preferred …

Content truncated to 500 words