TCU at SemEval-2022 Task 8: A Stacking Ensemble Transformer Model for Multilingual News Article Similarity

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1202 - 1207 July 14-15, 2022 ©2022 Association for Computational Linguistics TCU at SemEval-2022 Task 8: A Stacking Ensemble Transformer Model for Multilingual News Article Similarity Xiang Luo1 ,Yanqing Niu1 and Boer Zhu2 1School of Mathematics and Statistics, South Central University for Nationalities, Wuhan, China 2School of Computer Science, South Central University for Nationalities, Wuhan, China Contact: niuyanqing@mail.scuec.edu.cn Abstract Previous studies focus on measuring the de- gree of similarity of texts by using traditional machine learning methods, such as Support Vector Regression (SVR). Based on Trans- formers, this paper describes our contribution to SemEval-2022 Task 8 Multilingual News Article Similarity. The similarity of multilin- gual news articles requires a regression predic- tion on the similarity of multilingual articles, rather than a classification for judging text sim- ilarity. This paper mainly describes the archi- tecture of the model and how to adjust the pa- rameters in the experiment and strengthen the generalization ability. In this paper, we imple- ment and construct different models through transformer-based models. We applied differ- ent transformer-based models, as well as en- semble them together by using ensemble learn- ing. To avoid the overfit, we focus on the ad- justment of parameters and the increase of gen- eralization ability in our experiments. In the last submitted contest, we achieve a score of 0.715 and rank the 21st place. 1 Introduction Providing computer the ability to understand the abstract meaning of real world is a fundamental tasks. Given a pair of news articles, this task seek to evaluate the semantic similarity between them, which focuses on the real world-happenings cov- ered in the news articles. It’s a regression problem for measuring similarity of multilingual texts. Previous studies measured the similarity be- tween texts by using traditional machine learning methods, such as using Support Vector Regression (SVR) (Šarić et al., 2012). Recently many deep learning methods came out, such as pre-trained model. It had attracted the interest of researchers and had shown good result. For example, Explor- ing Bidirectional Encoder Representations from Transformers (BERT), XLNet and Robustly opti- mized BERT approach (RoBERTa) and finally got a good ranking (Yang et al., 2020). And a new hybridized approach using Weighted Fine-Tuned BERT Feature extraction with Siamese Bi-LSTM model has been implemented. It is employed for determining question pair sets using Semantic-text- similarity from Quora dataset (Viji and Revathy, 2022). These novel deep learning methods have performed well. In this study, we explored some transformer- based models. We had employed BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), A Lite BERT for Self-supervised Learning of Language Representations (ALBERT) (Lan et al., 2020), Dis- tilBERT (Sanh et al., 2020; Gou et al., 2021), be- sides this, we used them as base models, merged them together with ensemble learning, and the pre- diction result is used as a new training set, and then SVR is used as the meta model. The train- ing set generated by the base model is …

Content truncated to 500 words