Cross-lingual Few-Shot Learning on Unseen Languages

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 57–67 November 20–23, 2022. ©2022 Association for Computational Linguistics 57 Leveraging Key Information Modeling to Improve Less-Data Constrained News Headline Generation via Duality Fine-Tuning Zhuoxuan Jiang†, Lingfeng Qiao†, Di Yin†, Shanshan Feng‡, Bo Ren§ †Tencent Youtu Lab, Shanghai, China ‡Harbin Institute of Technology, Shenzhen, China §Tencent Youtu Lab, Hefei, China jzhx@pku.edu.cn, {leafqiao,endymecyyin,timren}@tencent.com, victor_fengss@foxmail.com Abstract Recent language generative models are mostly trained on large-scale datasets, while in some real scenarios, the training datasets are often expensive to obtain and would be small-scale. In this paper we investigate the challenging task of less-data constrained generation, especially when the generated news headlines are short yet expected by readers to keep readable and in- formative simultaneously. We highlight the key information modeling task and propose a novel duality fine-tuning method by formally defining the probabilistic duality constraints between key information prediction and headline gen- eration tasks. The proposed method can cap- ture more information from limited data, build connections between separate tasks, and is suit- able for less-data constrained generation tasks. Furthermore, the method can leverage various pre-trained generative regimes, e.g., autoregres- sive and encoder-decoder models. We conduct extensive experiments to demonstrate that our method is effective and efficient to achieve im- proved performance in terms of language mod- eling metric and informativeness correctness metric on two public datasets. 1 Introduction In an age of information explosion, headline gen- eration becomes one fundamental application in the natural language process (NLP) field (Tan et al., 2017; Li et al., 2021). Currently, the headline gener- ation is usually regarded as a special case of general text summarization. Therefore, many cutting-edge techniques based on pre-trained models and fine- tuning methods can be directly adapted by feeding headline generation datasets (Zhang et al., 2020b; Gu et al., 2020). Actually, compared with those textual summaries, headline generation aims at gen- erating only one sentence or a piece of short texts given a long document (e.g., a news article). It is challenging to guarantee the generated headline readable and informative at the same time, which is important to attract or inform readers especially for news domain (Matsumaru et al., 2020). Recently, some works find that neglecting the key information would degrade the performance of generative models which only consider captur- ing natural language (Nan et al., 2021b). Then many works about modeling different kinds of key information have been studied to enhance the infor- mation correctness of generative summaries. For example, overlapping salient words between source document and target summary (Li et al., 2020), key- words (Li et al., 2018), key phrases (Mao et al., 2020) and named entities (Nan et al., 2021a) are involved to design generative models. However, those works are mostly either trained on large-scale datasets or targets for long summaries (Ao et al., 2021). In some real applications, it is expensive to obtain massive labeled data. …

Content truncated to 500 words