Construction and Evaluation of Japanese Sentence-BERT Models

Construction and Evaluation of Japanese Sentence-BERT Models Naoki Shibayama Hiroyuki Shinnou Ibaraki University, Ibaraki, Japan {21nd303a, hiroyuki.shinnou.0828}@vc.ibaraki.ac.jp Abstract Sentence-BERT is model which based on “Bidirectional Encoder Representations from Transformers” (BERT) for building sentence embedding. This model has abilities for se- mantic analysis similar to BERT; however, the processing need not be online like in BERT. Therefore, it is also effective for similar sen- tence searches. No Japanese sentence-BERT model has been released in the right for- mat. Here, we built six Japanese sentence- BERT models with Japanese Stanford natu- ral language inference (JSNLI) released at Ky- oto University and six public Japanese BERT models. Furthermore, we proposed two eval- uation methods for the sentence-BERT mod- els and evaluated the six Japanese sentence- BERT models using the ratio of the in-class dispersion to out-of-class dispersions and ac- curacy of classification tasks using k-nearest neighbor (k-NN) classifier. As a result, two sentence-BERT models recorded higher per- formance: the model which was built from Tohoku BERT and the National Institute of Information and Communications Technology (NICT) BERT. 1 Introduction Sentence-BERT (Reimers and Gurevych, 2019) is a specialized BERT (Devlin et al., 2018) for build- ing sentence embedding. Normal sentence-BERT builds embedding for the input sequences of token IDs using the averages of output embeddings from BERT. The best feature of sentence-BERT is that the processing need not be online. A similar sen- tence search task requires suitable sentence embed- ding to measure similarity of two sentences; how- ever, embeddings made from simple models, such as a bag of words, cannot measure the similarity of meanings. While BERT overcomes this drawback, it can process cross-encoders to solve a task requir- ing a pair of sentences. Although BERT is not used from a viewpoint of processing time, the sentence- BERT processes as a bi-encoder; therefore, the prob- lem of processing time, as in the case of BERT, does not arise. Sentence-BERT can be also used for tasks which requires similarity of sentence meanings, and sentence-BERT has high effectiveness (Ndukwe et al., 2020)(Shirafuji et al., 2020)(Gencoglu, 2020). However, there is no proper pretrained model for Japanese sentence-BERT. Therefore, we propose a Japanese sentence-BERT that requires a Japanese- based pretrained BERT and large natural language inference (NLI) dataset. Here, we use the Japanse Stanford natural language inference (JSNLI) re- leaseatby Kyoto University as our large NLI dataset. Furthermore, we used six Japanese pretrained BERT to build and evaluate six sentence-BERTs: Ky- otoUniv. BERT, Stockmark BERT, Sentence-Piece BERT, Tohoku BERT, National Institute of Infor- mation and Communications Technology (NICT) BERT, and Laboro BERT. Evaluating the sentence-BERT requires setting tasks needed for the sentence-BERT and measur- ing the performances of the tasks. The appropriate tasks for evaluating sentence-BERT are tasks such as searching similar sentence and judging similar- ity between two sentences. However, cases suffice where preparing datasets of those tasks are impossi- ble. In this work, we devised two evaluation meth- ods to evaluate sentence-BERT. One of the methods embeds the title sentences of each document from dataset …

Content truncated to 500 words