Continuous Decomposition of Granularity for Neural Paraphrase Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 6369–6378 October 12–17, 2022. 6369 Continuous Decomposition of Granularity for Neural Paraphrase Generation Xiaodong Gu1, Zhaowei Zhang1, Sang-Woo Lee2, Kang Min Yoo2, Jung-Woo Ha2 1 School of Software, Shanghai Jiao Tong University 2 NAVER AI Lab {xiaodong.gu,andy_zhangzw}@sjtu.edu.cn {sang.woo.lee,kangmin.yoo,jungwoo.ha}@navercorp.com, Abstract While Transformers have had significant suc- cess in paragraph generation, they treat sen- tences as linear sequences of tokens and of- ten neglect their hierarchical information. Prior work has shown that decomposing the levels of granularity (e.g., word, phrase, or sentence) for input tokens has produced substantial im- provements, suggesting the possibility of en- hancing Transformers via more fine-grained modeling of granularity. In this work, we pro- pose continuous decomposition of granularity for neural paraphrase generation (C-DNPG). In order to efficiently incorporate granularity into sentence encoding, C-DNPG introduces a granularity-aware attention (GA-Attention) mechanism which extends the multi-head self- attention with: 1) a granularity head that auto- matically infers the hierarchical structure of a sentence by neurally estimating the granularity level of each input token; and 2) two novel at- tention masks, namely, granularity resonance and granularity scope, to efficiently encode granularity into attention. Experiments on two benchmarks, including Quora question pairs and Twitter URLs have shown that C-DNPG outperforms baseline models by a remarkable margin and achieves the state-of-the-art results in terms of many metrics. Qualitative analy- sis reveals that C-DNPG indeed captures fine- grained levels of granularity with effectiveness. 1 Introduction With the continued success in NLP tasks (Vaswani et al., 2017), Transformer has been the mainstream neural architecture for paraphrase generation (Li et al., 2019; Kazemnejad et al., 2020; Guo et al., 2021; Hosking et al., 2022; Goyal and Durrett, 2020). The core component of Transformer is the self-attention network (SAN) (Vaswani et al., 2017) which computes sentence representations at each position by baking representations over all other positions in a parallel way. Despite their effective- ness, Transformer has been shown to be limited in Text What is the reason for World War II ? decomposition 1 What is the reason for World War II ? decomposition 2 What is the reason for World War II ? decomposition 3 What is the reason for World War II ? decomposition 4 What is the reason for World War II ? decomposition 5 What is the reason for World War II ? ⇓ Levels of granularity (marked as superscripts): What1 is1 the2 reason3 of2 World4 War4 II5 ? Table 1: A motivation example of multi-granularity text decomposition. The given sentence can be decomposed according to 5 increasing levels of granularity, each cor- responding to a partition of the sentence into a template (blue) at a specific level together with details (orange). Each row with colored text denotes a level of granu- larity where the blue words are in the sentence level (templates) and the remaining words are in the phrase level (details). The bottom half shows the level of gran- ularity for each word according to the decomposition. We use integer …

Content truncated to 500 words