FinBART: A Pre-trained Seq2seq Language Model for Chinese Financial Tasks

FinBART: A Pre-trained Seq2seq Language Model for Chinese Financial Tasks Hongyuan Dong†, Wanxiang Che∗,†, Xiaoyu He‡, Guidong Zheng‡, Junjie Wen‡ †Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China ‡China Merchants Bank AILAB {hydong, car}@ir.hit.edu.cn {hexiaoyu42, zhengguidong, wenjunjieee}@cmbchina.com Abstract Pretrained language models are making a more profound impact on our lives than ever before. They exhibit promising performance on a variety of general domain Natural Language Process- ing (NLP) tasks. However, few work focuses on Chinese financial NLP tasks, which comprise a significant portion of social communication. To this end, we propose FinBART, a pretrained seq2seq language model for Chinese financial communication tasks. Experiments show that FinBART outperforms baseline models on a series of downstream tasks including text classifica- tion, sequence labeling and text generation. We further pretrain the model on customer service corpora, and results show that our model outperforms baseline models and achieves promising performance on various real world customer service text mining tasks. 1 Introduction From making investment decisions to managing personal finances, financial text data is crucial in helping individuals and organizations stay up-to-date with the latest financial trends. With the development of information technology, the volume of available financial information looms so large that the use of ma- chine learning methods to facilitate financial text processing becomes increasingly important. However, financial text data differs from general domain data as it contains a large number of technical terms and financial concepts, which require abundant expert knowledge to analyze and annotate. As a result, the application of powerful deep learning technology in financial text processing tasks is limited. In this work, we apply the fast-developing pretraining techniques to Chinese financial domain. Al- though a number of works attempt to adapt transformer-based [1] pretrained language models to financial text [2, 3, 4], few studies focus on the application on Chinese financial text data. What’s worse, these works adopt transformer encoder architecture, leading to limited ability in modeling the dependency be- tween future tokens and poor performance in text generation tasks. To this end, we propose FinBART, which is an encoder-decoder transformer language model pretrained on Chinese financial corpora. Fin- BART extracts bi-directional semantic information with its encoder, and generates text with the decoder part. In this way, FinBART is capable of tackling both understanding and generation tasks in Chinese financial domain. In real world scenarios of the financial industry, customer service is mainly conducted through text interaction. Therefore, there is a great demand for text analysis tools. Customer service corpora differ from financial ones for their sample text is highly colloquial. Although FinBART learns professional knowledge from the financial field, it cannot handle customer service tasks as well as financial ones without adaptation. To this end, we further pretrain FinBART model with customer service corpora, which are comprised of customers’ queries about financial services. Further pretraining on customer service corpora adapts FinBART to text interaction scenarios, enabling FinBART to capture accurate semantic information of customer queries. In summary, the main contributions of this work …

Content truncated to 500 words