Cross-Policy Compliance Detection via Question Answering

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8622–8632 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8622 Cross-Policy Compliance Detection via Question Answering Marzieh Saeidi Facebook AI marzieh@fb.com Majid Yazdani Facebook AI myazdani@fb.com Andreas Vlachos University of Cambridge Facebook AI av308@cam.ac.uk Abstract Policy compliance detection is the task of en- suring that a scenario conforms to a policy (e.g. a claim is valid according to government rules or a post in an online platform conforms to community guidelines). This task has been previously instantiated as a form of textual en- tailment, which results in poor accuracy due to the complexity of the policies. In this paper we propose to address policy compliance detec- tion via decomposing it into question answer- ing, where questions check whether the condi- tions stated in the policy apply to the scenario, and an expression tree combines the answers to obtain the label. Despite the initial upfront annotation cost, we demonstrate that this ap- proach results in better accuracy, especially in the cross-policy setup where the policies dur- ing testing are unseen in training. In addition, it allows us to use existing question answering models pre-trained on existing large datasets. Finally, it explicitly identifies the information missing from a scenario in case policy com- pliance cannot be determined. We conduct our experiments using a recent dataset con- sisting of government policies, which we aug- ment with expert annotations and find that the cost of annotating question answering decom- position is largely offset by improved inter- annotator agreement and speed. 1 Introduction Policy compliance detection is the task of ensuring that a scenario conforms to a policy. It is a task that occurs in various contexts, e.g. in ensuring correct application of government policies (Saeidi et al., 2018; Holzenberger et al., 2020), enforcing community guidelines on social media platforms (Waseem and Hovy, 2016) or the resolution of legal cases (Zhong et al., 2020), etc. While the task can be modelled as a text-pair classification similar to textual entailment (Dagan 1Policy source can be found here https://www.gov. uk/housing-benefit/how-to-claim Q0: Is it up to 13 weeks in advance? Q1: Is it up to 17 weeks in advance? Q2: Are you aged 60 or over? Q3: Are you moving? ➤ Claiming in advance and backdating AND OR AND Q3 Q0 Q1 Q2 You can claim in advance by up to 13 weeks (or 17 weeks if you’re aged 60 or over), if you’re moving. You won’t usually get any money before you move. Scenario: My partner and I will be moving shortly. I had my 40th birthday last week. Policy Decomposition (A3 and (A0 or (A1 and A2) )) = NEI A0: NEI, A1: NEI, A2: No, A3: Yes Direct Label: NEI Label via Decomposition: NEI = Not Enough Information/Unresolved Figure 1: A policy with its decomposition (questions and expression tree), scenario and label.1 et al., 2009; Bowman et al., 2015), i.e. whether a scenario complies with a certain policy, this often results in …

Content truncated to 500 words