Infusing Finetuning with Semantic Dependencies

Infusing Finetuning with Semantic Dependencies Zhaofeng Wu♠ Hao Peng♠ Noah A. Smith♠♦ ♠Paul G. Allen School of Computer Science & Engineering, University of Washington ♦Allen Institute for Artificial Intelligence {zfw7,hapeng,nasmith}@cs.washington.edu Abstract For natural language processing systems, two kinds of evidence support the use of text representations from neural language models ‘‘pretrained’’ on large unannotated corpora: performance on application-inspired bench- marks (Peters et al., 2018, inter alia), and the emergence of syntactic abstractions in those representations (Tenney et al., 2019, inter alia). On the other hand, the lack of grounded supervision calls into question how well these representations can ever capture meaning (Bender and Koller, 2020). We apply novel probes to recent language models— specifically focusing on predicate-argument structure as operationalized by semantic de- pendencies (Ivanova et al., 2012)—and find that, unlike syntax, semantics is not brought to the surface by today’s pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning, yielding benefits to natural language understanding (NLU) tasks in the GLUE benchmark. This approach de- monstrates the potential for general-purpose (rather than task-specific) linguistic supervi- sion, above and beyond conventional pretrain- ing and finetuning. Several diagnostics help to localize the benefits of our approach.1 1 Introduction The past decade has seen a paradigm shift in how NLP systems are built, summarized as follows: • Before, general-purpose linguistic modules (e.g., part-of-speech taggers, word-sense disambiguators, and many kinds of parsers) were constructed using supervised learning from linguistic datasets. These were often 1https://github.com/ZhaofengWu/SIFT. applied as preprocessing to text as part of larger systems for information extraction, question answering, and other applications. • Today, general-purpose representation learn- ing is carried out on large, unannotated corpora—effectively a kind of unsupervised learning known as ‘‘pretraining’’—and then the representations are ‘‘finetuned’’ on application-specific datasets using conven- tional end-to-end neural network methods. The newer paradigm encourages an emphasis on corpus curation, scaling up pretraining, and translation of end-user applications into trainable ‘‘tasks,’’ purporting to automate most of the la- bor requiring experts (linguistic theory construc- tion, annotation of data, and computational model design). Apart from performance improvements on virtually every task explored in the NLP lit- erature, a body of evidence from probing studies has shown that pretraining brings linguistic abs- tractions to the surface, without explicit super- vision (Liu et al., 2019a; Tenney et al., 2019; Hewitt and Manning, 2019; Goldberg, 2019, inter alia). There are, however, reasons to pause. First, some have argued from first principles that learn- ing mappings from form to meaning is hard from forms alone (Bender and Koller, 2020).2 Second, probing studies have focused more heavily on syntax than on semantics (i.e., mapping of forms to abstractions of meaning intended by people speaking in the world). Tenney et al. (2019) noted that the BERT model (Devlin et al., 2019) offeredmore to syntactic tasks like constituent and dependency relation labeling than semantic ones 2In fact, Bender and Koller (2020) argued that this is impossible for grounded semantics. Our probing analysis, along with recent efforts (Kovaleva et …

Content truncated to 500 words