On the Limits of Learning to Actively Learn Semantic Representations

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 452–462 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 452 On the Limits of Learning to Actively Learn Semantic Representations Omri Koshorek1 Gabriel Stanovsky2,4 Yichu Zhou3 Vivek Srikumar3 Jonathan Berant1,2 1Tel-Aviv University, 2Allen Institute for AI 3The University of Utah, 4University of Washington {omri.koshorek,joberant}@cs.tau.ac.il gabis@allenai.org, {flyaway,svivek}@cs.utah.edu Abstract One of the goals of natural language under- standing is to develop models that map sen- tences into meaning representations. How- ever, training such models requires expensive annotation of complex structures, which hin- ders their adoption. Learning to actively-learn (LTAL) is a recent paradigm for reducing the amount of labeled data by learning a policy that selects which samples should be labeled. In this work, we examine LTAL for learning semantic representations, such as QA-SRL. We show that even an oracle policy that is al- lowed to pick examples that maximize perfor- mance on the test set (and constitutes an up- per bound on the potential of LTAL), does not substantially improve performance compared to a random policy. We investigate factors that could explain this finding and show that a dis- tinguishing characteristic of successful appli- cations of LTAL is the interaction between op- timization and the oracle policy selection pro- cess. In successful applications of LTAL, the examples selected by the oracle policy do not substantially depend on the optimization pro- cedure, while in our setup the stochastic nature of optimization strongly affects the examples selected by the oracle. We conclude that the current applicability of LTAL for improving data efficiency in learning semantic meaning representations is limited. 1 Introduction The task of mapping a natural language sentence into a semantic representation, that is, a structure that represents its meaning, is one of the core goals of natural language processing. This goal has led to the creation of many general-purpose for- malisms for representing the structure of language, such as semantic role labeling (SRL; Palmer et al., 2005), semantic dependencies (SDP; Oepen et al., 2014), abstract meaning representation (AMR; Banarescu et al., 2013), universal conceptual cog- nitive annotation (UCCA; Abend and Rappoport, 2013), question-answer driven SRL (QA-SRL; He et al., 2015), and universal dependencies (Nivre et al., 2016), as well as domain-specific semantic representations for particular users in fields such as biology (Kim et al., 2009; N´edellec et al., 2013; Berant et al., 2014) and material science (Mysore et al., 2017; Kim et al., 2019). Currently, the dominant paradigm for building models that predict such representations is super- vised learning, which requires annotating thou- sands of sentences with their correct structured representation, usually by experts. This arduous data collection is the main bottleneck for building parsers for different users in new domains. Past work has proposed directions for accelerat- ing data collection and improving data efficiency through multi-task learning across different rep- resentations (Stanovsky and Dagan, 2018; Hersh- covich et al., 2018), or having non-experts anno- tate sentences in natural language (He et al., 2015, 2016). One of …

Content truncated to 500 words