Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 818–829 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 818 Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling †Ernie Chang, †Vera Demberg, *Alex Marin †Dept. of Language Science and Technology, Saarland University *Microsoft Corporation, Redmond, WA cychang@coli.uni-saarland.de Abstract Neural natural language generation (NLG) and understanding (NLU) models are data-hungry and require massive amounts of annotated data to be competitive. Recent frameworks ad- dress this bottleneck with generative models that synthesize weak labels at scale, where a small amount of training labels are expert- curated and the rest of the data is automatically annotated. We follow that approach, by au- tomatically constructing a large-scale weakly- labeled data with a fine-tuned GPT-2, and em- ploy a semi-supervised framework to jointly train the NLG and NLU models. The pro- posed framework adapts the parameter updates to the models according to the estimated label- quality. On both the E2E and Weather bench- marks, we show that this weakly supervised training paradigm is an effective approach un- der low resource scenarios with as little as 10 data instances, and outperforming benchmark systems on both datasets when 100% of train- ing data is used. 1 Introduction Natural language generation (NLG) is the task that transforms meaning representations (MR) into nat- ural language descriptions (Reiter and Dale, 2000; Barzilay and Lapata, 2005); while natural language understanding (NLU) is the opposite process where text is converted into MR (Zhang and Wang, 2016). These two processes can thus constrain each other – recent exploration of the duality of neural natu- ral language generation (NLG) and understanding (NLU) has led to successful semi-supervised learn- ing techniques where both labeled and unlabeled data can be used for training (Su et al., 2020; Tseng et al., 2020; Schmitt and Sch¨utze, 2019; Qader et al., 2019; Su et al., 2020). Standard supervised learning for NLG and NLU depends on the access to labeled training data – a major bottleneck in developing new applications. Human Annotator Weak Annotator Clean Text labels Noisy text labels Unlabeled MR x x x x x x x x x x x x x x x x x Figure 1: Training scenario: Each × represents a labeled data instance. The goal is to learn both from few human- labeled instances (inner) and large amounts of weakly labeled data (outer). In particular, neural methods require a large anno- tated dataset for each specific task. The collection process is often prohibitively expensive, especially when specialized domain expertise is required. On the other hand, learning with weak supervision from noisy labels offers a potential solution as it automatically builds imperfect training sets from low cost labeling rules or pretrained models (Zhou, 2018; Ratner et al., 2017; Fries et al., 2020). Fur- ther, labeled data and large unlabeled data can be utilized in semi-supervised learning (Lample et al., 2017; Tseng et al., 2020), as a way to jointly im- prove …

Content truncated to 500 words