Joint Learning of Pre-Trained and Random Units for Domain Adaptation in Part-of-Speech Tagging

Proceedings of NAACL-HLT 2019, pages 4107–4112 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4107 Joint Learning of Pre-Trained and Random Units for Domain Adaptation in Part-of-Speech Tagging Sara Meftah∗, Youssef Tamaazousti∓, Nasredine Semmar∗, Hassane Essafi∗, Fatiha Sadat+ ∗CEA, LIST, LASTI, France ∓MIT, CSAIL, USA +UQ `AM, Montr´eal, Canada {firstname.lastname}@cea.fr , ytamaaz@mit.edu , sadat.fatiha@uqam.ca Abstract Fine-tuning neural networks is widely used to transfer valuable knowledge from high- resource to low-resource domains. In a stan- dard fine-tuning scheme, source and target problems are trained using the same architec- ture. Although capable of adapting to new do- mains, pre-trained units struggle with learning uncommon target-specific patterns. In this pa- per, we propose to augment the target-network with normalised, weighted and randomly ini- tialised units that beget a better adaptation while maintaining the valuable source knowl- edge. Our experiments on POS tagging of so- cial media texts (Tweets domain) demonstrate that our method achieves state-of-the-art per- formances on 3 commonly used datasets. 1 Introduction POS tagging is a sequence labelling problem, that consists on assigning to each sentence’ word, its disambiguated POS tag (e.g., Pronoun, Noun) in the phrasal context in which the word is used. Such information is useful for higher-level ap- plications, such as machine-translation (Niehues and Cho, 2017) or cross-lingual information re- trieval (Semmar et al., 2006, 2008). One of the best approaches for POS tag- ging of social media text (Meftah et al., 2018a), is transfer-learning, which relies on a neural- network learned on a source-dataset with suffi- cient annotated data, then further adapted to the problem of interest (target-dataset). While this approach is known to be very effective (Zen- naki et al., 2019), because it takes benefit from pre-trained neurons, it has one main drawback by design. Indeed, it has been shown in computer- vision (Zhou et al., 2018a) that, when fine-tuning on scenes a model pre-trained on objects, it is the neuron firing on the white dog object that became highly sensitive to the white waterfall scene. Sim- ply said, pre-trained neurons are biased by what they have learned in the source-dataset. This is Figure 1: Given a word representation xi, a BiLSTM (Φ) models the sequence, and a FC layer (Ψ) performs classification. In standard fine-tuning, the units are pre-trained on a large source-dataset then adapted to the target one. In this work, we propose to add ran- domly initialised units (green branch) and jointly adapt them with pre-trained ones (gray branch). An element- wise sum is further applied to merge the two branches. Before merging, we balance the different behaviours of pre-trained and random units, using an independent normalisation (N ). Finally we let the network learn which of pre-trained or random neurons are more suited for every class, by adding learnable weighting vectors (u and v initialised with 1-values) on the FC layers. also the case on NLP (see experiments). Con- sequently, pre-trained units struggle with learn- ing patterns specific to the target-dataset (e.g., “wanna” or “gonna” in the …

Content truncated to 500 words