Question Answering for Privacy Policies: Combining Computational and Legal Perspectives

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4947–4958, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4947 Question Answering for Privacy Policies: Combining Computational and Legal Perspectives Abhilasha Ravichander ♦ Alan Black ♦ Shomir Wilson ♥ Thomas Norton♠ Norman Sadeh♦ ♦Carnegie Mellon University, Pittsburgh, PA ♥Penn State University, University Park, PA ♠Fordham Law School, New York, NY {aravicha, awb, sadeh}@cs.cmu.edu {shomir}@psu.edu, {tnorton1}@law.fordham.edu Abstract Privacy policies are long and complex doc- uments that are difficult for users to read and understand, and yet, they have legal ef- fects on how user data is collected, managed and used. Ideally, we would like to empower users to inform themselves about issues that matter to them, and enable them to selec- tive explore those issues. We present PRIVA- CYQA, a corpus consisting of 1750 questions about the privacy policies of mobile appli- cations, and over 3500 expert annotations of relevant answers. We observe that a strong neural baseline underperforms human per- formance by almost 0.3 F1 on PRIVACYQA, suggesting considerable room for improve- ment for future systems. Further, we use this dataset to shed light on challenges to ques- tion answerability, with domain-general im- plications for any question answering sys- tem. The PRIVACYQA corpus offers a chal- lenging corpus for question answering, with genuine real-world utility. 1 Introduction Privacy policies are the documents which dis- close the ways in which a company gathers, uses, shares and manages a user’s data. As le- gal documents, they function using the princi- ple of notice and choice (Federal Trade Commis- sion, 1998), where companies post their policies, and theoretically, users read the policies and de- cide to use a company’s products or services only if they find the conditions outlined in its pri- vacy policy acceptable. Many legal jurisdictions around the world accept this framework, includ- ing the United States and the European Union (Patrick, 1980; OECD, 2004). However, the legiti- macy of this framework depends upon users ac- tually reading and understanding privacy poli- cies to determine whether company practices are acceptable to them (Reidenberg et al., 2015). Figure 1: Examples of privacy-related questions users ask, drawn from two mobile applications: Keep1and Viber.2Policy evidence represents sentences in the privacy policy that are relevant for determining the answer to the user’s question.3 In practice this is seldom the case (Cate, 2010; Cranor, 2012; Schaub et al., 2015; Gluck et al., 2016; Jain et al., 2016; US Federal Trade Commis- sion et al., 2012; McDonald and Cranor, 2008). This is further complicated by the highly indi- vidual and nuanced compromises that users are willing to make with their data (Leon et al., 2015), discouraging a ‘one-size-fits-all’ approach to no- tice of data practices in privacy documents. With devices constantly monitoring our envi- ronment, including our personal space and our bodies, lack of awareness of how our data is be- ing used easily leads to problematic situations where users are …

Content truncated to 500 words