RECIPE: Applying Open Domain Question Answering to Privacy Policies

Proceedings of the Workshop on Machine Reading for Question Answering, pages 71–77 Melbourne, Australia, July 19, 2018. c⃝2018 Association for Computational Linguistics 71 RECIPE: Applying Open Domain Question Answering to Privacy Policies Yan Shvartzshnaider New York University and Princeton Univerity yansh@nyu.edu Ananth Balashankar New York University ananth@nyu.edu Thomas Wies New York University wies@cs.nyu.edu Lakshminarayanan Subramanian New York University lakshmi@cs.nyu.edu Abstract We describe our experiences in us- ing an open domain question answering model (Chen et al., 2017) to evaluate an out-of-domain QA task of assisting in analyzing privacy policies of companies. Specifically, Relevant CI Parameters Ex- tractor (RECIPE) seeks to answer ques- tions posed by the theory of contextual integrity (CI) regarding the information flows described in the privacy statements. These questions have a simple syntactic structure and the answers are factoids or descriptive in nature. The model achieved an F1 score of 72.33, but we noticed that combining the results of this model with a neural dependency parser based approach yields a significantly higher F1 score of 92.35 compared to manual annotations. This indicates that future work which in- corporates signals from parsing like NLP tasks more explicitly can generalize better on out-of-domain tasks. 1 Introduction Open domain question answering approaches of- fer a promising glimpse into a future in which ma- chines are able to perform sophisticated cognitive tasks on behalf of a human. Recent advances in deep neural networks applied to reading compre- hension and document retrieval (Chen et al., 2017; Wang et al., 2017) have achieved competitive re- sults for answering questions based on a large and diverse corpus of documents. These models are trained on Wikipedia text and are capable of an- swering various factoid and descriptive questions. Specifically, the distribution of the interrogative words used, lexical and syntactic variations, rea- soning across multiple sentences and ambiguous statements in such Wikipedia datasets (Ryu et al., 2014; Rajpurkar et al., 2016) results in a robust QA model. Motivated by their success, we set to apply one such model (Chen et al., 2017) to evaluate an out- of-domain QA task to assist in analysis and un- derstanding of privacy policies. The text in pri- vacy policies is notoriously cumbersome, confus- ing and hard to comprehend even for legal ex- perts (Reidenberg et al., 2015). So when it comes to reading privacy policies, users often miss im- portant information or skip reading them alto- gether. Past efforts have applied NLP (Sathyendra et al., 2017, 2016; Evans et al., 2017), Machine Learning (Ammar et al., 2012) and crowdsourc- ing (Wilson et al., 2016b) techniques to identify important and relevant privacy statements. How- ever, identifying paragraphs that mention sensitive information in privacy policies only takes us half- way because we still need to understand who col- lects that information, who receives it, and under what conditions the collection happens. To sup- port this finer-grained analysis, we present a case for using an machine comprehension model for answering questions posed by the theory of con- textual integrity (CI) (Nissenbaum, 2010). To …

Content truncated to 500 words