SubjQA: A Dataset for Subjectivity and Review Comprehension

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5480–5494, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5480 SUBJQA: A Dataset for Subjectivity and Review Comprehension Johannes Bjerva1,3∗ Nikita Bhutani2∗ Behzad Golshan2 Wang-Chiew Tan2 Isabelle Augenstein1 1Department of Computer Science, University of Copenhagen 2Megagon Labs, Mountain View 3Department of Computer Science, Aalborg University jbjerva@cs.aau.dk, augenstein@di.ku.dk {nikita,behzad,wangchiew}@megagon.ai Abstract Subjectivity is the expression of internal opin- ions or beliefs which cannot be objectively observed or verified, and has been shown to be important for sentiment analysis and word- sense disambiguation. Furthermore, subjectiv- ity is an important aspect of user-generated data. In spite of this, subjectivity has not been investigated in contexts where such data is widespread, such as in question answering (QA). We develop a new dataset which allows us to investigate this relationship. We find that subjectivity is an important feature in the case of QA, albeit with more intricate inter- actions between subjectivity and QA perfor- mance than found in previous work on senti- ment analysis. For instance, a subjective ques- tion may or may not be associated with a sub- jective answer. We release an English QA dataset (SUBJQA) based on customer reviews, containing subjectivity annotations for ques- tions and answer spans across 6 domains. 1 Introduction Subjectivity is ubiquitous in our use of language (Banfield, 1982; Quirk et al., 1985; Wiebe et al., 1999; Benamara et al., 2017), and is therefore an important aspect to consider in Natural Language Processing (NLP). For example, subjectivity can be associated with different senses of the same word. BOILING is objective in the context of hot water, but subjective in the context of a person boiling with anger (Wiebe and Mihalcea, 2006). The same applies to sentences in discourse contexts (Pang and Lee, 2004). While early work has shown sub- jectivity to be an important feature for low-level tasks such as word-sense disambiguation and sen- timent analysis, subjectivity in NLP has not been explored in many contexts where it is prevalent. ∗JB and NB contributed equally to this work. In recent years, there has been renewed interest in areas of NLP for which subjectivity is impor- tant, and a specific topic of interest is question answering (QA). This includes work on aspect ex- traction (Poria et al., 2016), opinion mining (Sun et al., 2017) and community QA (Gupta et al., 2019). Many QA systems are based on representa- tion learning architectures (e.g. Devlin et al. (2018); Radford et al. (2018)) that are typically trained on factual, encyclopedic knowledge such as Wikipedia or books. It is unclear if these architectures can han- dle subjective statements such as those that appear in reviews. The interactions between QA and subjectivity are even more relevant today as users’ natural search criteria in many domains, including prod- ucts and services, have become increasingly sub- jective. According to McAuley and Yang (2016a), around 20% of product queries were labeled as be- ing ‘subjective’ by workers. Their questions can of- ten be answered by …

Content truncated to 500 words