Pre-train or Annotate? Domain Adaptation with a Constrained Budget

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 498–517 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 498 Multitask Semi-Supervised Learning for Class-Imbalanced Discourse Classification Alexander Spangher, Jonathan May University of Southern California {spangher, jonmay}@usc.edu Sz-rung Shiang, Lingjia Deng Bloomberg {sshiang, ldeng43} @bloomberg.net Abstract As labeling schemas evolve over time, small differences can render datasets following older schemas unusable. This prevents researchers from building on top of previous annotation work and results in the existence, in discourse learning in particular, of many small class- imbalanced datasets. In this work, we show that a multitask learning approach can com- bine discourse datasets from similar and di- verse domains to improve discourse classifica- tion. We show an improvement of 4.9% Micro F1-score over current state-of-the-art bench- marks on the NewsDiscourse dataset, one of the largest discourse datasets recently pub- lished, due in part to label correlations across tasks, which improve performance for under- represented classes. We also offer an exten- sive review of additional techniques proposed to address resource-poor problems in NLP, and show that none of these approaches can im- prove classification accuracy in our setting1. 1 Introduction Learning the discourse structure of a text has been shown to be helpful for diverse tasks such as event extraction (Choubey et al., 2020), sentiment analy- sis (Chenlo et al., 2014), natural language genera- tion (Celikyilmaz et al., 2020), summarization (Lu et al., 2019; Isonuma et al., 2019), storyline discov- ery (Rehm et al., 2019), and even misinformation detection (Abbas, 2020; Zhou et al., 2020). However, even as recent advances in NLP allow us to achieve impressive results across a variety of tasks, discourse learning, a supervised learning task, faces the following challenges: (1) discourse datasets tend to be very class-imbalanced.2 (2) Discourse learning is a complex task: human anno- tators require training and conferencing to achieve 1For code and data, see: https://github.com/ alex2awesome/multitask-news-discourse. 2For example, of Penn Discourse Tree-Bank’s 48 classes, the top 24 are on average 25 times more common than the bottom 24 (Prasad et al., 2008). moderate agreement (Das et al., 2017). (3) Dis- course learning tends to be resource-poor, as an- notation complexities make large-scale data col- lection challenging (Table 1). Compounding the problem, a schema often evolves across different annotation efforts, preventing the compilation of smaller datasets into larger ones.3 We observe, however, that certain discourse schemata appear to offer complementary informa- tion. For example, Penn Discourse and Rhetor- ical Structure Theory Treebanks offer intrasen- tential, low-level discourse information (Prasad et al., 2008; Carlson et al., 2003), while news dis- course schemas offer intersentential, high-level, domain-specific discourse information (Choubey et al., 2020; Yarlott et al., 2018). Inspired by Col- lobert and Weston (2008)’s finding that lower-level NLP tasks (e.g. part of speech tagging) could aid higher-level tasks (e.g. semantic role labeling), we hypothesize that a multitask approach incor- porating multiple discourse datasets can address the challenges listed above. Specifically, by intro- ducing complementary information from auxiliary discourse tasks, we can increase performance …

Content truncated to 500 words