PolicyQA: A Reading Comprehension Dataset for Privacy Policies

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 743–749 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 743 PolicyQA: A Reading Comprehension Dataset for Privacy Policies Wasi Uddin Ahmad∗ University of California, Los Angeles wasiahmad@ucla.edu Jianfeng Chi∗ University of Virginia jc6ub@virginia.edu Yuan Tian University of Virginia yuant@virginia.edu Kai-Wei Chang University of California, Los Angeles kwchang.cs@ucla.edu Abstract Privacy policy documents are long and ver- bose. A question answering (QA) system can assist users in finding the information that is relevant and important to them. Prior studies in this domain frame the QA task as retrieving the most relevant text segment or a list of sen- tences from the policy document given a ques- tion. On the contrary, we argue that providing users with a short text span from policy docu- ments reduces the burden of searching the tar- get information from a lengthy text segment. In this paper, we present PolicyQA, a dataset that contains 25,017 reading comprehension style examples curated from an existing cor- pus of 115 website privacy policies. PolicyQA provides 714 human-annotated questions writ- ten for a wide range of privacy practices. We evaluate two existing neural QA models and perform rigorous analysis to reveal the advan- tages and challenges offered by PolicyQA. 1 Introduction Security and privacy policy documents describe how an entity collects, maintains, uses, and shares users’ information. Users need to read the privacy policies of the websites they visit or the mobile applications they use and know about their privacy practices that are pertinent to them. However, prior works suggested that people do not read privacy policies because they are long and complicated (McDonald and Cranor, 2008), and confusing (Rei- denberg et al., 2016). Hence, giving users access to a question answering system to search for answers from long and verbose policy documents can help them better understand their rights. In recent years, we have witnessed noteworthy progress in developing question answering (QA) systems with a colossal effort to benchmark high- quality, large-scale datasets for a few application ∗Equal contribution. Website: Amazon.com Information You Give Us: We receive and store any information you enter on our Web site or give us in any other way. Click here to see ... Question. How do you collect my information? information you enter on our Web site Promotional Offers: Sometimes we send offers to selected groups of Amazon.com customers on behalf of other businesses. When we do this, we do not give that business your name and address. If you do not want to receive such offers, ... Question. Is my information shared with others? we do not give that business your name and address Table 1: Question-answer pairs that we collect from OPP-115 (Wilson et al., 2016a) dataset. The evidence spans are highlighted in color and they are used to form the question-answer pairs. domains (e.g., Wikipedia, news articles). However, annotating large-scale QA datasets for domains such as security and privacy is challenging as it requires expert annotators (e.g., law students). Due …

Content truncated to 500 words