DomBERT: Domain-oriented Language Model for Aspect-based Sentiment Analysis

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1725–1731 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1725 DomBERT: Domain-oriented Language Model for Aspect-based Sentiment Analysis Hu Xu1, Bing Liu1, Lei Shu1 and Philip S. Yu1,2 1Department of Computer Science, University of Illinois at Chicago, Chicago, IL, USA 2Institute for Data Science, Tsinghua University, Beijing, China {hxu48, liub, lshu3, psyu}@uic.edu Abstract This paper focuses on learning domain- oriented language models driven by end tasks, which aims to combine the worlds of both general-purpose language models (such as ELMo and BERT) and domain-specific lan- guage understanding. We propose DomBERT, an extension of BERT to learn from both in- domain corpus and relevant domain corpora. This helps in learning domain language mod- els with low-resources. Experiments are con- ducted on an assortment of tasks in aspect- based sentiment analysis (ABSA), demonstrat- ing promising results. 1 1 Introduction Pre-trained language models (LMs) (Peters et al., 2018; Radford et al., 2018, 2019; Devlin et al., 2019) aim to learn general (or mixed-domain) knowledge for end tasks. Recent studies (Xu et al., 2019; Gururangan et al., 2020) show that learning domain-specific LMs are equally important. This is because the training corpus of general LMs is out-of-domain for end tasks in a particular domain and, more importantly, because general LMs may not capture the long-tailed and underrepresented domain details (Xu et al., 2018). An intuitive ex- ample related to corpus of aspect-based sentiment analysis (ABSA) can be found in Table 1, where all masked words sky, water, idea, screen and picture can appear in a mixed-domain corpus. A general- purpose LM may favor frequent examples and ig- nore long-tailed choices in certain domains. In contrast, although domain-specific LMs can capture fine-grained domain details, they may suf- fer from insufficient training corpus (Gururan- gan et al., 2020) to strengthen general knowledge within a domain. To this end, we propose a domain- 1The code will be released on https://github.com/ howardhsu/BERT-for-RRC-ABSA. Example Domain The [MASK] is clear . The sky is clear . Astronomy [Irrelevant Domain] The water is clear . Liquids [Irrelevant Domain] The idea is clear . Concepts [Irrelevant Domain] The screen is clear . Desktop [Relevant Domain] The picture is clear . Laptop [Target Domain] Table 1: Multiple choices to recover a masked token (an aspect in ABSA) for different domains: a target do- main needs more examples from a relevant domain. oriented learning task that aims to combine the ben- efits of both general and domain-specific worlds: Domain-oriented Learning: Given a target do- main t and a set of diverse source domains S = {s1, s2, . . . }, perform (language model) learning that focusing on t and all its relevant domains in S. This learning task resolves the issues in both general and domain-specific worlds. On one hand, the training of LM does not need to focus on un- related domains anymore (e.g., Books is one big domain but not very related to laptop); on the other hand, although …

Content truncated to 500 words