A Comparative Study of Generative Pre-trained Transformer-based Models for Chinese Slogan Generation of Crowdfunding

XFEVER: Exploring Fact Verification across Languages Yi-Chen Chang1∗ Canasai Kruengkrai2 Junichi Yamagishi2 1National Tsing Hua University, Taiwan yichen@nlplab.cc 2National Institute of Informatics, Japan {canasai,jyamagishi}@nii.ac.jp Abstract This paper introduces the Cross-lingual Fact Extraction and VERification (XFEVER) dataset designed for benchmarking the fact ver- ification models across different languages. We constructed it by translating the claim and ev- idence texts of the Fact Extraction and VERi- fication (FEVER) dataset released by Thorne et al. (2018) into six languages. The training and development sets were translated using ma- chine translation, whereas the test set includes texts translated by professional translators and machine-translated texts. Using the XFEVER dataset, two cross-lingual fact verification sce- narios, zero-shot learning and translate-train learning, are defined, and baseline models for each scenario are also proposed in this paper. Experimental results show that the multilingual language model can be used to build fact verifi- cation models in different languages efficiently. However, the performance varies by language and is somewhat inferior to the English case. We also found that we can effectively mitigate model miscalibration by considering the predic- tion similarity between the English and target languages.1 Keywords: cross-lingual fact verification, pre- trained language models 1 Introduction Automated fact verification is a part of the fact- checking task, verifying that a given claim is valid against a database of textual sources. can be formulated as a classification task, tak-It ing the claim and associated evidence as input and determining whether the given evidence sup- ports the claim. Deep learning is used to build ∗This work was conducted during the author’s internship under National Institute of Informatics, Japan. 1The XFEVER dataset, code, and model check- points are available at https://github.com/ nii-yamagishilab/xfever. classifiers for this purpose, but deep models are data-hungry and require massive amounts of la- beled data. The Fact Extraction and VERifica- tion (FEVER) database (Thorne et al., 2018) is known as a well-resourced English database that enables us to build large networks, but building a database of the same scale as FEVER from scratch for each language is significantly time-consuming and costly. Our main question in this paper is: Can we build fact-checking models for other languages without huge costs? In this work, we hypothesize that facts are facts regardless of languages. Suppose we have a per- fect translator to translate English text into other languages without missing or changing informa- tion in the original texts. The relationship between a specific claim-evidence pair in the source lan- guage, which is the output of the fact verifica- tion model, should be the same even if they are translated into another target language as shown in Figure 1. Using this hypothesis, we construct a new Cross-lingual Fact Extraction and VERifi- cation (XFEVER) dataset by automatically trans- lating the claim and evidence texts of the FEVER dataset into five other languages: Spanish, French, Indonesian, Japanese, and Chinese. These lan- guages cover several language families, including isolated languages such as Japanese. In addition to the machine-translated texts, a set of texts writ- ten and …

Content truncated to 500 words