Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 201–210 May 2-6, 2023 ©2023 Association for Computational Linguistics Retrieval Enhanced Data Augmentation for Question Answering on Privacy Policies Md Rizwan Parvez§, Jianfeng Chi†, Wasi Uddin Ahmad§, Yuan Tian§, Kai-Wei Chang§ §University of California, Los Angeles, †University of Virginia §{rizwan,wasiahmad,yuant,kwchang}@ucla.edu, †jc6ub@virgina.edu Abstract Prior studies in privacy policies frame the ques- tion answering (QA) task as identifying the most relevant text segment or a list of sentences from a policy document given a user query. Ex- isting labeled datasets are heavily imbalanced (only a few relevant segments), limiting the QA performance in this domain. In this paper, we develop a data augmentation framework based on ensembling retriever models that captures the relevant text segments from unlabeled pol- icy documents and expand the positive exam- ples in the training set. In addition, to improve the diversity and quality of the augmented data, we leverage multiple pre-trained language mod- els (LMs) and cascade them with noise reduc- tion filter models. Using our augmented data on the PrivacyQA benchmark, we elevate the existing baseline by a large margin (10% F1) and achieve a new state-of-the-art F1 score of 50%. Our ablation studies provide further in- sights into the effectiveness of our approach. 1 Introduction Privacy policies describe how service providers col- lect, manage, and use their users’ data. Understand- ing them is crucial for users as they can determine if the conditions outlined are acceptable. Policy documents, however, are lengthy, verbose, equivo- cal, and hard to understand (McDonald and Cranor, 2008; Reidenberg et al., 2016). Consequently, they are often ignored and skipped by users (Commis- sion et al., 2012; Gluck et al., 2016). Building question answering (QA) systems for privacy policies is a stepping stone to allow users to ask questions about their rights. Prior works (Harkous et al., 2018; Ravichander et al., 2019) framed the QA task as a sentence selection task, es- sentially a binary classification task that identifies if a policy text segment is relevant to a question. Since policy documents consist of many sentences and typically a few are relevant to a question,1 the 1PrivacyQA (Ravichander et al., 2019) dataset has 1,350 Segmented policy document S (s1) We do not sell or rent your personal infor- mation to third parties for their direct marketing purposes without your explicit consent. (sn) ...We will not let any other person, including sellers and buyers, contact you, other than through your ... Queries I annotating the red segment as irrelevant (i1) How does Fiverr protect freelancers’ personal information? (i2) What type of identifiable infor- mation is passed between users on the platform? Queries R annotating the red segment as relevant (r1) What are the app’s permissions? (r2) What type of permissions does the app require? Queries D that annotators disagree about relevance (d1) Do you sell my information to third parties? (d2) Is my information sold to any third parties? Table 1: QA (sentence selection) from a policy …

Content truncated to 500 words