Intent Classification and Slot Filling for Privacy Policies

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4402–4417 August 1–6, 2021. ©2021 Association for Computational Linguistics 4402 Intent Classification and Slot Filling for Privacy Policies Wasi Uddin Ahmad†∗, Jianfeng Chi‡∗, Tu Le‡ Thomas Norton§, Yuan Tian‡, Kai-Wei Chang† †University of California, Los Angeles, ‡University of Virginia, §Fordham University †{wasiahmad, kwchang}@cs.ucla.edu ‡{jc6ub,tnl6wk,yuant}@virginia.edu §tnorton1@law.fordham.edu Abstract Understanding privacy policies is crucial for users as it empowers them to learn about the in- formation that matters to them. Sentences writ- ten in a privacy policy document explain pri- vacy practices, and the constituent text spans convey further specific information about that practice. We refer to predicting the privacy practice explained in a sentence as intent clas- sification and identifying the text spans shar- ing specific information as slot filling. In this work, we propose PolicyIE, an English corpus consisting of 5,250 intent and 11,788 slot an- notations spanning 31 privacy policies of web- sites and mobile applications. PolicyIE corpus is a challenging real-world benchmark with limited labeled examples reflecting the cost of collecting large-scale annotations from do- main experts. We present two alternative neu- ral approaches as baselines, (1) intent classifi- cation and slot filling as a joint sequence tag- ging and (2) modeling them as a sequence-to- sequence (Seq2Seq) learning task. The exper- iment results show that both approaches per- form comparably in intent classification, while the Seq2Seq method outperforms the sequence tagging approach in slot filling by a large mar- gin. We perform a detailed error analysis to reveal the challenges of the proposed corpus. 1 Introduction Privacy policies inform users about how a service provider collects, uses, and maintains the users’ in- formation. The service providers collect the users’ data via their websites or mobile applications and analyze them for various purposes. The users’ data often contain sensitive information; therefore, the users must know how their information will be used, maintained, and protected from unauthorized and unlawful use. Privacy policies are meant to explain all these use cases in detail. This makes ∗Equal contribution. Listed by alphabetical order. privacy policies often very long, complicated, and confusing (McDonald and Cranor, 2008; Reiden- berg et al., 2016). As a result, users do not tend to read privacy policies (Commission et al., 2012; Gluck et al.; Marotta-Wurgler, 2015), leading to un- desirable consequences. For example, users might not be aware of their data being sold to third-party advertisers even if they have given their consent to the service providers to use their services in return. Therefore, automating information extraction from verbose privacy policies can help users understand their rights and make informed decisions. In recent years, we have seen substantial efforts to utilize natural language processing (NLP) tech- niques to automate privacy policy analysis. In lit- erature, information extraction from policy doc- uments is formulated as text classification (Wil- son et al., 2016a; Harkous et al., 2018; Zimmeck et al., 2019), text alignment (Liu et al., 2014; Ra- …

Content truncated to 500 words