Evaluation of Pretrained BERT Model by Using Sentence Clustering

Evaluation of Pretrained BERT Model by Using Sentence Clustering Naoki Shibayama Rui Cao Jing Bai Wen Ma Hiroyuki Shinnou Ibaraki University, Department of Computer and Information Sciences 4-12-1 Nakanarusawa, Hitachi, Ibaraki JAPAN 316-8511 {19nm714t, 18nd305g, 19nd301r, 19nd302h, hiroyuki.shinnou.0828} @vc.ibaraki.ac.jp Abstract For evaluation of pre-trained models like bidi- rectional encoder representations from trans- formers (BERT), task-based approaches are frequently adopted and there is a possibility that meta parameters for fine-tuning influence results of the evaluations. However, task- based approaches for languages, except En- glish, have a problem- there is no common dataset for their evaluation. Hence, evaluating pre-trained models for these languages with task-based approaches is challenging. In this work, we evaluate Japanese pre-trained BERT models with CLS token. We input labeled sen- tences to models, get CLS token embeddings, and calculate scores from in-class and out- of-class dispersions, which can be calculated from embeddings and labels of sentences. Ex- periment results show that a model released by Laboro.AI Inc. is the best Japanese pre- trained BERT model. Meanwhile, the results of evaluation with sentence clustering are dif- ferent from those of evaluations that are based on fill mask task. 1 Introduction BERT (Devlin et al., 2019) is a high-performance pre-training model. It helped in the improve- ment of the performance of natural language pro- cessing tasks. Generally, task-based approaches were adopted for evaluating pre-training models like BERT. In English language, a dataset for task-based evaluation, such as the general language understand- ing evaluation (GLUE) (Wang et al., 2018), can be used, and it is easy to compare models. However, when a pre-trained model is fine-tuned for task- based evaluation, meta parameters for fine-tuning may influence scores of the model. Hence, task- based evaluation with fine-tuning has a possibility of biased evaluation. Also, there is no common task- based dataset for languages except English, so it is challenging to compare pre-trained models for other languages. In this work, we evaluate Japanese pre-trained BERT models using CLS token embeddings in out- puts of target models. CLS token embedding can be regarded as an input sentence embedding, and mod- els can be rated with evaluating embeddings itself. However, how to evaluate sentence embeddings is also challenging. Here, we use clustering to evalu- ate sentence embeddings. Also, we prepare sets of sentences sorted by genre and use BERT models to get embeddings of each sentence. Then, we cluster those embeddings and evaluate models with cluster- ing score. 2 Related Works Generally, a task-based approach for evaluation is adopted to compare and evaluate pre-trained mod- els like BERT. Although this simple method requires data for evaluation, it consists of the following 3 steps: 1. Solve a task with pre-trained model A and get its accuracy. 2. Solve this same task with pre-trained model B and get its accuracy. 3. Compare the accuracies and evaluate models A and B. The GLUE can be used for English, but there is no common dataset for other languages, so we have to prepare the dataset for evaluation ourselves. There …

Content truncated to 500 words