TruthfulQA: Measuring How Models Mimic Human Falsehoods

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 270 - 284 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Modeling U.S. State-Level Policies by Extracting Winners and Losers from Legislative Texts Maryam Davoodi Purdue University mdavoodi@purdue.edu Eric Waltenburg Purdue University ewaltenb@purdue.edu Dan Goldwasser Purdue University dgoldwas@purdue.edu Abstract Decisions on state-level policies have a deep effect on many aspects of our everyday life, such as health-care and education access. However, there is little understanding of how these policies and decisions are being formed in the legislative process. We take a data- driven approach by decoding the impact of leg- islation on relevant stakeholders (e.g., teach- ers in education bills) to understand legisla- tors’ decision-making process and votes. We build a new dataset for multiple US states that interconnects multiple sources of data includ- ing bills, stakeholders, legislators, and money donors. Next, we develop a textual graph- based model to embed and analyze state bills. Our model predicts winners/losers of bills and then utilizes them to better determine the leg- islative body’s vote breakdown according to demographic/ideological criteria, e.g., gender. 1 Introduction State-level legislation is the cornerstone of national policies and has long-lasting effects on residents of US states. Thus, decoding the processes that shape state bills is crucial yet involved. State leg- islatures vote on 23 times more bills than Federal legislatures, exceeding 120K bills per year (King, 2019). In addition, these state bills cover a broader range of local problems as each state possesses lawmaking power effective within its boundaries. E.g., the State of Washington Health Care Com- mittee addresses health service issues including li- censing and regulation of health care facilities and providers. Moreover, it regulates pharmacies, phar- maceutical drugs, state public health programs, and private/public insurance markets (House, 2021). We argue that recent NLP architectures can provide new insights into the state-level legislative efforts. In particular, contextualized graph and text embedding can better represent policies within and across states via a shared political context. However, most of the prior efforts are focused on analyzing congressional bills with traditional techniques, e.g., (Gerrish and Blei, 2011, 2012). A few state-level studies (Eidelman et al., 2018; Davoodi et al., 2020) took great steps in predicting the progression of state bills towards a vote on the floor and the breakdown of votes based on demographic metrics (e.g., gender). But their main downside is they evaluate policies in a limited context and do not capture cross-state patterns. Medical malpractice actions. Permits a patient to bring an action against a health care provider without submitting the complaint to the medical review board if: (1) the amount of the claim is not more than $15,000; (2) the cause of action is based on the removal of the wrong body part Cleavage Yea Nay Figure 1: A health bill leading to voting cleavage based on the party metric primarily due to its specific losers (red) and winners (green). Winners-Losers analysis. In this work, we take a new …

Content truncated to 500 words