Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4432 - 4441 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation Mingzhe Li1,2,∗, Xiexiong Lin3,∗, Xiuying Chen4, Jinxiong Chang3, Qishen Zhang3, Feng Wang3, Taifeng Wang3, Zhongyi Liu3, Wei Chu3, Dongyan Zhao1,2,†, Rui Yan5,† 1Wangxuan Institute of Computer Technology, Peking University, Beijing, China 2Center for Data Science, AAIS, Peking University, Beijing, China 3 Ant Group 4 Computational Bioscience Reseach Center, KAUST 5 Gaoling School of Artificial Intelligence, Renmin University of China li_mingzhe@pku.edu.cn, xiexiong.lxx@antfin.com Abstract Contrastive learning has achieved impressive success in generation tasks to militate the “ex- posure bias” problem and discriminatively ex- ploit the different quality of references. Exist- ing works mostly focus on contrastive learn- ing on the instance-level without discriminat- ing the contribution of each word, while key- words are the gist of the text and dominant the constrained mapping relationships. Hence, in this work, we propose a hierarchical contrastive learning mechanism, which can unify hybrid granularities semantic meaning in the input text. Concretely, we first propose a keyword graph via contrastive correlations of positive-negative pairs to iteratively polish the keyword repre- sentations. Then, we construct intra-contrasts within instance-level and keyword-level, where we assume words are sampled nodes from a sentence distribution. Finally, to bridge the gap between independent contrast levels and tackle the common contrast vanishing problem, we propose an inter-contrast mechanism that measures the discrepancy between contrastive keyword nodes respectively to the instance dis- tribution. Experiments demonstrate that our model outperforms competitive baselines on paraphrasing, dialogue generation, and story- telling tasks. 1 Introduction Generation tasks such as storytelling, paraphras- ing, and dialogue generation aim at learning a cer- tain correlation between text pairs that maps an arbitrary-length input to another arbitrary-length output. Traditional methods are mostly trained with “teacher forcing” and lead to an “exposure bias” problem (Schmidt, 2019). Incorporating the gen- eration method with contrastive learning achieved impressive performance on tackling such issues, which takes an extra consideration of synthetic neg- ative samples contrastively (Lee et al., 2021). ∗ Equal Contribution † Corresponding authors: Rui Yan and Dongyan Zhao min keyword graph max astrophysics imposter Input: What are the best books on cosmology? Input Semantic Space cosmology astrophysics cosmology astrophysics Target output: Which is the best book for cosmology? Negative output: Which is the best introductory book on astrophysics? latent mapping cosmology Output Semantic Space keyword mapping Figure 1: The semantic meaning of the sentence “what are the best books on cosmology?” would be greatly changed if the keyword “cosmology” is changed to “as- trophysic”. Existing contrastive mechanisms are mainly fo- cused on the instance level (Lee et al., 2021; Cai et al., 2020). However, word-level information is also of great importance. Take the case shown in the upper part of Figure 1 for example, the keyword covers the gist of the input text and determines the embedding space of the text. The text …

Content truncated to 500 words