Feature Adaptation of Pre-Trained Language Models across Languages and Domains with Robust Self-Training
29877Authors
nan
Abstract
AbstractAdapting pre-trained language models (PrLMs) (e.g., BERT) to new domains has gained much attention recently. Instead of fine-tuning PrLMs as done in most previous work, we investigate how to adapt the features of PrLMs to new domains without fine-tuning. We explore unsupervised domain adaptation (UDA) in this paper. With the features from PrLMs, we adapt the models trained with labeled data from the source domain to the unlabeled target domain. Self-training is widely used for UDA, and it predicts pseudo labels on the target domain data for training. However, the predicted pseudo labels inevitably include noise, which will negatively affect training a robust model. To improve the robustness of self-training, in this paper we present class-aware feature self-distillation (CFd) to learn discriminative features from PrLMs, in which PrLM features are self-distilled into a feature adaptation module and the features from the same class are more tightly clustered. We further extend CFd to a cross-language setting, in which language discrepancy is studied. Experiments on two monolingual and multilingual Amazon review datasets show that CFd can consistently improve the performance of self-training in cross-domain and cross-language settings.
Keywords
Full Content
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 806–822, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 806 Where Are You? Localization from Embodied Dialog Meera Hahn1 Jacob Krantz2 Dhruv Batra1,3 Devi Parikh1,3 James M. Rehg1 Stefan Lee2 Peter Anderson1∗ 1Georgia Institute of Technology 2Oregon State University 3Facebook AI Research (FAIR) {mhahn30, dbatra, parikh, rehg}@gatech.edu {krantzja, leestef}@oregonstate.edu pjand@google.com Abstract We present WHERE ARE YOU? (WAY), a dataset of ∼6k dialogs in which two humans – an Observer and a Locator – complete a co- operative localization task. The Observer is spawned at random in a 3D environment and can navigate from first-person views while an- swering questions from the Locator. The Lo- cator must localize the Observer in a detailed top-down map by asking questions and giving instructions. Based on this dataset, we define three challenging tasks: Localization from Em- bodied Dialog or LED (localizing the Observer from dialog history), Embodied Visual Dialog (modeling the Observer), and Cooperative Lo- calization (modeling both agents). In this pa- per, we focus on the LED task – providing a strong baseline model with detailed ablations characterizing both dataset biases and the im- portance of various modeling choices. Our best model achieves 32.7% success at identi- fying the Observer’s location within 3m in un- seen buildings, vs. 70.4% for human Locators. 1 Introduction Imagine getting lost in a new building while trying to visit a friend who lives or works there. Unsure of exactly where you are, you call your friend and start describing your surroundings (‘I’m standing near a big blue couch in what looks like a lounge. There are a set of wooden double doors opposite the entrance.’) and navigating in response to their questions (‘If you go through those doors, are you in a hallway with a workout room to the right?’). After a few rounds of dialog, your friend who is familiar with the building will hopefully know your location. Success at this cooperative task requires goal-driven questioning based on your friend’s un- derstanding of the environment, unambiguous an- swers communicating observations via language, ∗Now at Google. Locator Observer I am in a kitchen with a wood dining table with orange placemats. Can you describe the area you are in? No I am on side opposite the kitchen. Are you standing on the side table near the kitchen? I am next to a brown circular table with gray chairs. Are you in the kitchen or the living room? Can you describe where you are? I’m at the edge of the kitchen near the white counter Locator Observer Locator Observer Figure 1: LED Task: The Locator has a top-down map of the building and is trying to localize the Observer by asking questions and giving instructions. The Ob- server has a first person view and may navigate while responding to the Locator. The turn-taking dialog ends when the Locator predicts the Observer’s position. and active perception and navigation to investigate the environment and seek out discriminative …
Content truncated to 500 words
Labeled Text
AbstractAdapting pre-trained language models (PrLMs) (e.g., BERT) to new domains has gained much attention recently. Instead of fine-tuning PrLMs as done in most previous work, we investigate how to adapt the features of PrLMs to new domains without fine-tuning. We explore unsupervised domain adaptation (UDA) in this paper. With the features from PrLMs, we adapt the models trained with labeled data from the source domain to the unlabeled target domain. Self-training is widely used for UDA, and it predicts pseudo labels on the target domain data for training. However, the predicted pseudo labels inevitably include noise, which will negatively affect training a robust model. To improve the robustness of self-training, in this paper we present class-aware feature self-distillation (CFd) to learn discriminative features from PrLMs, in which PrLM features are self-distilled into a feature adaptation module and the features from the same class are more tightly clustered. We further extend CFd to a cross-language setting, in which language discrepancy is studied. Experiments on two monolingual and multilingual Amazon review datasets show that CFd can consistently improve the performance of self-training in cross-domain and cross-language settings.
Publication Details
- Paper ID:
- 29877
- Year:
- 2020
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan