Optimizing Domain Specificity of Transformer-based Language Models for Extractive Summarization of Financial News Articles in Korean

Optimizing Domain Specificity of Transformer-based Language Models for Extractive Summarization of Financial News Articles in Korean Huije Lee Wonsuk Yang ChaeHun Park Hoyun Song Eugene Jang Jong C. Park∗ School of Computing Korea Advanced Institute of Science and Technology {jae4258,derrick0511,ddehun,hysong,eugenej,park}@nlp.kaist.ac.kr Abstract Frequent usage of complex expressions with numbers and of the terms that require domain knowledge makes it more difficult to com- prehend and summarize financial news arti- cles than that of other daily news articles. We present a transformer-based model for the au- tomatic summarization of the financial news articles in Korean and address related issues, and in particular analyze the interplay between the domain of the dataset used for pre-training and that for fine-tuning. We find that the sum- marization model performs much better when the two coincide, even when they are different from that of the target task, which is the finan- cial domain in our work. 1 Introduction It has been widely acknowledged that some pre- trained language models can be effective for tasks with little training data, as pre-trained models are often able to achieve moderate (and sometimes sat- isfactory) performance, even without a large dataset that had been commonly employed to train neural models before the introduction of such pre-trained language models (Radford et al., 2019; Zhang et al., 2020). It has also been acknowledged that such pre-trained language models are effective for low resource languages (Conneau and Lample, 2019). Also, language-agnostic tokenization methods such as Byte-Pair Encoding (Sennrich et al., 2016) and SentencePiece (Kudo and Richardson, 2018) have ∗ Corresponding author been introduced, making it possible to fine-tune lan- guage models without language-specific tokenizers. Many of the recently introduced language mod- els are based on transformers (Vaswani et al., 2017). Among them, pre-trained language models gained much attention due to their advanced perfor- mance on various tasks. Specialized language mod- els have also been introduced to domains such as biomedicine (Lee et al., 2020), clinics (Huang et al., 2019), and finance (Liu et al., 2020). When we use such a pre-trained language model for a domain-specific task, the language model can be expected to perform better if the model is pre- trained specifically for the domain at hand. How- ever, it has been recently noted that it is non-trivial to determine whether systems that use a pre-trained model trained on a domain-specific corpus may out- perform those that use a model pre-trained on a gen- eral domain corpus (Lee et al., 2020; Gururangan et al., 2020) In this work, we address the summarization task for financial news articles in Korean. We follow ex- tractive summarization rather than abstractive sum- marization, as the latter can mistakenly use some numbers or terminology that do not appear in the original document (Kryscinski et al., 2019). In con- trast, extractive summarization only selects sen- tences (or phrases) from the original document, without such ‘hallucinations’ (Koay et al., 2020). We analyze the effectiveness of a model based on the domain specificities of the datasets that are used for …

Content truncated to 500 words