Nikkei at SemEval-2022 Task 8: Exploring BERT-based Bi-Encoder Approach for Pairwise Multilingual News Article Similarity

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1208 - 1214 July 14-15, 2022 ©2022 Association for Computational Linguistics Nikkei at SemEval-2022 Task 8: Exploring BERT-based Bi-Encoder Approach for Pairwise Multilingual News Article Similarity Shotaro Ishihara Hono Shirai Nikkei, Inc. Chiyoda, Tokyo, Japan {shotaro.ishihara,hono.shirai}@nex.nikkei.com Abstract This paper describes our system in SemEval- 2022 Task 8, where participants were required to predict the similarity of two multilingual news articles. In the task of pairwise sentence and document scoring, there are two main ap- proaches: Cross-Encoder, which inputs pairs of texts into a single encoder, and Bi-Encoder, which encodes each input independently. The former method often achieves higher perfor- mance, but the latter gave us a better result in SemEval-2022 Task 8. This paper presents our exploration of BERT-based Bi-Encoder ap- proach for this task, and there are several find- ings such as pretrained models, pooling meth- ods, translation, data separation, and the num- ber of tokens. The weighted average ensemble of the four models achieved the competitive result and ranked in the top 12. 1 Introduction Measuring sentence and document similarity is a task that has been studied for many years in the field of natural language processing. One of the applications is to identify whether news articles address the same subject. If news articles can be properly clustered, they can be used for a wide range of purposes, such as recommendation and displaying related articles. SemEval-2022 Task 8 attempts to tackle this task with multilingual news articles (Chen et al., 2022). Nowadays, it is common for this kind of tasks to use transformer-based models like BERT (Devlin et al., 2019). There are several research directions, including post-processing (Li et al., 2020; Wang and Kuo, 2020), unsupervised learning (Zhang et al., 2020; Tiyajamorn et al., 2021), and su- pervised learning (Reimers and Gurevych, 2019; Thakur et al., 2021; Feng et al., 2020; Jiang et al., 2022). Here, since a labeled dataset was provided, we decided to use a supervised learning approach. It is standard practice to combine two sentences as input when dealing with pairwise similarity of Figure 1: The architectures of Cross-Encoder and Bi- Encoder. The input of the Cross-Encoder architecture is two sentences joined by SEP, and through BERT and a pooling layer, the regressor outputs a score. In the Bi-Encoder architecture, each sentence is transformed by BERT and pooling layers, and the score is calculated through interaction of the two vectors. sentences in a supervised learning approach with BERT (Lin et al., 2021; Reimers and Gurevych, 2019). In contrast, we chose an approach that em- beds each sentence separately. Figure 1 shows these two approaches, named Cross-Encoder and Bi-Encoder to follow the previous research (Thakur et al., 2021). This paper describes our system in SemEval- 2022 Task 8. First, we explain the experimental results by adopting the Bi-Encoder architecture rather than Cross-Encoder. We also present the following research questions: 1) which pretrained model works well when dealing with multilingual news articles, 2) what kind of …

Content truncated to 500 words