Flexible Generation from Fragmentary Linguistic Input

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8176 - 8196 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Flexible Generation from Fragmentary Linguistic Input Peng Qian Roger P. Levy Department of Brain and Cognitive Sciences Massachusetts Institute of Technology {pqian,rplevy}@mit.edu Abstract The dominant paradigm for high-performance models in novel NLP tasks today is direct specialization for the task via training from scratch or fine-tuning large pre-trained mod- els. But does direct specialization capture how humans approach novel language tasks? We hypothesize that human performance is bet- ter characterized by flexible inference through composition of basic computational motifs available to the human language user. To test this hypothesis, we formulate a set of novel fragmentary text completion tasks, and com- pare the behavior of three direct-specialization models against a new model we introduce, GibbsComplete, which composes two basic computational motifs central to contemporary models: masked and autoregressive word pre- diction. We conduct three types of evalua- tion: human judgments of completion quality, satisfaction of syntactic constraints imposed by the input fragment, and similarity to hu- man behavior in the structural statistics of the completions. With no task-specific parame- ter tuning, GibbsComplete performs compara- bly to direct-specialization models in the first two evaluations, and outperforms all direct- specialization models in the third evaluation. These results support our hypothesis that hu- man behavior in novel language tasks and envi- ronments may be better characterized by flexi- ble composition of basic computational motifs rather than by direct specialization. 1 Introduction Representation learning has tremendously bene- fited engineering for language comprehension and generation systems. General frameworks such as encoder-decoder or auto-regressive model can be flexibly applied to a diverse range of problems. With scaled models, enormous data, and well- chosen training objective functions, generative pre- training extracts useful and transferable informa- tion from unlabeled text data (Howard and Ruder, 2018; Liu et al., 2019b). These generic representa- tions can then be finetuned to quickly yield perfor- mant models directly specialized for downstream tasks (Howard and Ruder, 2018; Radford et al., 2019; Devlin et al., 2019, inter alia). While the broad idea of exploiting rich statistical informa- tion in general-purpose representations has proven practically effective for deep learning models since a decade ago (Erhan et al., 2010; Collobert et al., 2011), it remains an open question how well this “direct specialization” approach yields models that behave similarly to humans in flexible linguistic behavior in novel situations. Here we take this “direct specialization” ap- proach as a scientific hypothesis regarding the na- ture of linguistic knowledge and its flexible deploy- ment in the human mind: that the human capacity of processing linguistic information in novel tasks arises from directly specializing and reshaping a generic yet versatile representation. We contrast it with an alternative hypothesis: that flexible knowl- edge deployment reflects algorithmic composition of a constrained repertoire of simple, reusable infer- ence motifs. We depict these competing hypothe- ses in Figure 1. In the “compositional inference” hypothesis, the …

Content truncated to 500 words