Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding
4786Authors
nan
Abstract
AbstractGeneralized text representations are the foundation of many natural language understanding tasks. To fully utilize the different corpus, it is inevitable that models need to understand the relevance among them. However, many methods ignore the relevance and adopt a single-channel model (a coarse paradigm) directly for all tasks, which lacks enough rationality and interpretation. In addition, some existing works learn downstream tasks by stitches skill block (a fine paradigm), which might cause irrational results due to its redundancy and noise. In this work, we first analyze the task correlation through three different perspectives, , data property, manual design, and model-based relevance, based on which the similar tasks are grouped together. Then, we propose a hierarchical framework with a coarse-to-fine paradigm, with the bottom level shared to all the tasks, the mid-level divided to different groups, and the top-level assigned to each of the tasks. This allows our model to learn basic language properties from all tasks, boost performance on relevant tasks, and reduce the negative impact from irrelevant tasks. Our experiments on 13 benchmark datasets across five natural language understanding tasks demonstrate the superiority of our method.
Keywords
Full Content
Proceedings of the 29th International Conference on Computational Linguistics, pages 521–531 October 12–17, 2022. 521 Section-Aware Commonsense Knowledge-Grounded Dialogue Generation with Pre-trained Language Model Sixing Wu1, Ying Li2,3∗, Ping Xue2, Dawei Zhang1, Zhonghai Wu2,3 1School of Computer Science, Peking University, Beijing, China 2 School of Software and Microelectronics, Peking University, Beijing, China 3National Research Center of Software Engineering, Peking University, Beijing, China Abstract In knowledge-grounded dialogue generation, pre-trained language models (PLMs) can be ex- pected to deepen the fusing of dialogue context and knowledge because of their superior abil- ity of semantic understanding. Unlike adopt- ing the plain text knowledge, it is thorny to leverage the structural commonsense knowl- edge when using PLMs because most PLMs can only operate plain texts. Thus, linearizing commonsense knowledge facts into plan text is a compulsory trick. However, a dialogue is always aligned to a lot of retrieved fact candi- dates; as a result, the linearized text is always lengthy and then significantly increases the bur- den of using PLMs. To address this issue, we propose a novel two-stage framework SAKDP. In the first pre-screening stage, we use a rank- ing network PriorRanking to estimate the rel- evance of a retrieved knowledge fact. Thus, facts can be clustered into three sections of dif- ferent priorities. As priority decreases, the rel- evance decreases, and the number of included facts increases. In the next dialogue genera- tion stage, we use section-aware strategies to encode the linearized knowledge. The power- ful but expensive PLM is only used for a few facts in the higher priority sections, reaching the performance-efficiency balance. Both the automatic and human evaluation demonstrate the superior performance of this work. 1 Introduction Dialogue systems strive to facilitate human-like di- alogue responses (Chen et al., 2017). One essential precondition for generating high-quality dialogue is a sufficient cognition of the contextually-relevant knowledge besides the literal surface. The dialogue is grounded on both the given user query and the context-related knowledge (Yu et al., 2020). For ex- ample, given a query ‘Are you thirsty?’, rather than ‘Yes/No’, a meaningful response should have more ∗ Corresponding author: Ying Li, li.ying@pku.edu.cn. The email of the first author: wusixing@pku.edu.cn thirsty drink water buy a beer drink a person thirsty a person buy a beer drink water drink Causes Causes Requires NotDesires Linearization [CSK] thirsty Requires drink [CSK] a person NotDesires thirsty [CSK] thirsty Causes drink water [CSK] thirsty Causes buy a peer Structural Commonsense Facts (4 facts) Linearized Commonsense Facts (20 tokens) Figure 1: An example of linearizing knowledge. information, such as ‘Yes, and I’m going to drink some water.’ (context-related knowledge →thirsty causes drinking water). Seeking information from external sources is a feasible way to enhance the machine’s cogni- tion of knowledge (Dinan et al., 2019; Zhou et al., 2021b). As for the choice of knowledge source, the structural commonsense knowledge (Speer et al., 2017) is a proven option, which consists of a lot of knowledge facts that are frequently used in daily life. As shown in Figure 1, …
Content truncated to 500 words
Labeled Text
AbstractGeneralized text representations are the foundation of many natural language understanding tasks. To fully utilize the different corpus, it is inevitable that models need to understand the relevance among them. However, many methods ignore the relevance and adopt a single-channel model (a coarse paradigm) directly for all tasks, which lacks enough rationality and interpretation. In addition, some existing works learn downstream tasks by stitches skill block (a fine paradigm), which might cause irrational results due to its redundancy and noise. In this work, we first analyze the task correlation through three different perspectives, , data property, manual design, and model-based relevance, based on which the similar tasks are grouped together. Then, we propose a hierarchical framework with a coarse-to-fine paradigm, with the bottom level shared to all the tasks, the mid-level divided to different groups, and the top-level assigned to each of the tasks. This allows our model to learn basic language properties from all tasks, boost performance on relevant tasks, and reduce the negative impact from irrelevant tasks. Our experiments on 13 benchmark datasets across five natural language understanding tasks demonstrate the superiority of our method.
Publication Details
- Paper ID:
- 4786
- Year:
- 2022
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan