Low Resource Sequence Tagging using Sentence Reconstruction

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2692–2698 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2692 Low Resource Sequence Tagging using Sentence Reconstruction Tal Perl Tel Aviv University talperl@mail.tau.ac.il Sriram Chaudhury Wipro sriramchaudhury@gmail.com Raja Giryes Tel Aviv University raja@tauex.tau.ac.il Abstract This work revisits the task of training se- quence tagging models with limited resources using transfer learning. We investigate several proposed approaches introduced in recent works and suggest a new loss that relies on sentence reconstruction from normalized embeddings. Specifically, our method demonstrates how by adding decoding layer for sentence reconstruction,a we can improve the performance of various baselines. We show improved results on the CoNLL02 NER and UD 1.2 POS datasets and demonstrate the power of the method for transfer learning with low-resources achieving 0.6 F1 score in Dutch using only one sample from it. The code is publicly available at: https://github.com/tperl/Low- Resource-Sequence-Tagging-using-Sentence- Reconstruction. 1 Introduction The increased popularity of deep learning led to a giant leap in natural language processing (NLP). Tasks such as neural machine translation (Lample et al., 2018a; Gu et al., 2018), sentiment analysis (Patro et al., 2018) and question answering (Ran et al., 2019) achieved impressive results. A major limitation of deep learning is the need for huge amounts of training data. Thus, when deal- ing with low resource datasets, transfer learning is a common solution. A popular approach in NLP is training a language model for getting a good context-based word representation. Language mod- els such as Bert (Devlin et al., 2019), Roberta (Liu et al., 2019b), ELMO (Peters et al., 2018), and XL- net (Yang et al., 2019) that are trained on very large corpora, are used by the community for different NLP tasks. This “transfer-learning” across tasks within the same language relies on fine-tuning a lan- guage model for a specific task (Sun et al., 2019). This work focuses on transfer learning between different languages. Some approaches have been suggested for it. Yang et al. (2017) have proposed using joint training with a large dataset as a source and a small dataset as a target. Zou et al. (2018) have shown how by aligning sentence representa- tions using an adversarial loss, they were able to transfer knowledge between two languages. Contribution. This work analyzes the contri- bution of various techniques proposed for trans- fer learning between languages for the task of se- quence tagging. In particular, we evaluate joint training and adversarial learning. Moreover, we propose a novel regularization technique, namely, we add a reconstruction loss with ℓ2 normalization. We show that the addition of this loss improves the performance of various sequence tagging tasks when doing transfer learning. Our strategy shows promising results for train- ing models without being language-specific, which saves expensive labeling time. An important char- acteristic of our technique is its ability to provide good tagging in ”few-shot learning” (Fei-Fei et al., 2006). We achieve this result by adding to the small dataset, a larger corpus corresponding to …

Content truncated to 500 words