Transductive Learning of Neural Language Models for Syntactic and Semantic Analysis

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3665–3671, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3665 Transductive Learning of Neural Language Models for Syntactic and Semantic Analysis Hiroki Ouchi1,2 Jun Suzuki2,1 Kentaro Inui2,1 1 RIKEN Center for Advanced Intelligence Project 2 Tohoku University hiroki.ouchi@riken.jp, {jun.suzuki,inui}@ecei.tohoku.ac.jp Abstract In transductive learning, an unlabeled test set is used for model training. While this setting deviates from the common assumption of a completely unseen test set, it is applicable in many real-world scenarios, where the texts to be processed are known in advance. However, despite its practical advantages, transductive learning is underexplored in natural language processing. Here, we conduct an empirical study of transductive learning for neural mod- els and demonstrate its utility in syntactic and semantic tasks. Specifically, we fine-tune lan- guage models (LMs) on an unlabeled test set to obtain test-set-specific word representations. Through extensive experiments, we demon- strate that despite its simplicity, transductive LM fine-tuning consistently improves state-of- the-art neural models in both in-domain and out-of-domain settings. 1 Introduction In supervised learning, a model is trained on a training set and its generalization performance is evaluated on an unseen test set. In this setting, the model has no access to the test set during training. However, the assumption of a completely unseen test set is not always necessary. In many cases, certain aspects of the test set are already known at training time. For example, a company may want to annotate a large number of existing documents automatically (Section 3). In such a scenario, the texts to be processed are known in advance, and using the model trained on the texts themselves to process them can be more efficient. Using an un- labeled test set in this way is the key idea behind transductive learning. In transductive learning (Vapnik, 1998), an un- labeled test set is given in the training phase. That is, the inputs of the test set, i.e., the raw texts, can be used during training, but the labels are never used. In the test phase, the trained model is eval- uated on the same test set. Despite its practical advantages, transductive learning has received lit- tle attention in natural language processing (NLP). After the pioneering work of Joachims (1999), who proposed a transductive support vector ma- chine for text classification, transductive methods for linear models have been investigated in only a few tasks, such as lexical acquisition (Duh and Kirchhoff, 2006) and machine translation (Ueffing et al., 2007). In particular, transductive learning with neural networks is underexplored. Here, we investigate the impact of transductive learning on state-of-the-art neural models in syn- tactic and semantic tasks, namely syntactic chunk- ing and semantic role labeling (SRL). Specifically, inspired by recent findings that language model (LM)-based word representations yield large per- formance improvement (Devlin et al., 2019), we fine-tune Embeddings from Language Models (ELMo) (Peters et al., 2018) on an unlabeled …

Content truncated to 500 words