Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-text Rationales

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7103–7128 July 9-14, 2023 ©2023 Association for Computational Linguistics Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationales Brihi Joshi♣∗ Ziyi Liu♣∗ Sahana Ramnath♣ Aaron Chan♣ Zhewei Tong♢ Shaoliang Nie♠ Qifan Wang♠ Yejin Choi♦♥ Xiang Ren♣♥ ♣University of Southern California ♢ Tsinghua University ♠Meta AI ♥Allen Institute for Artificial Intelligence ♦ University of Washington {brihijos, zliu2803, sramnath, chanaaro, xiangren}@usc.edu , tzw19@mails.tsinghua.edu.cn {snie, wqfcr}@meta.com, yejin@cs.washington.edu Abstract Among the remarkable emergent capabilities of large language models (LMs) is free-text ra- tionalization; beyond a certain scale, large LMs are capable of generating seemingly useful ra- tionalizations, which in turn, can dramatically enhance their performances on leaderboards. This phenomenon raises a question: can ma- chine generated rationales also be useful for humans, especially when lay humans try to answer questions based on those machine ra- tionales? We observe that human utility of ex- isting rationales is far from satisfactory, and expensive to estimate with human studies. Ex- isting metrics like task performance of the LM generating the rationales, or similarity between generated and gold rationales are not good indi- cators of their human utility. While we observe that certain properties of rationales like con- ciseness and novelty are correlated with their human utility, estimating them without human involvement is challenging. We show that, by estimating a rationale’s helpfulness in answer- ing similar unseen instances, we can measure its human utility to a better extent. We also translate this finding into an automated score, GEN-U, that we propose, which can help im- prove LMs’ ability to generate rationales with better human utility, while maintaining most of its task performance. Lastly, we release all code and collected data with this project.1 1 Introduction In recent years, there has been a surge of interest in using language models (LMs) for human-AI col- laboration (Wiegreffe et al., 2022; You and Lowd, 2022). For example, LMs have played a large role in reducing human effort for dataset creation (Bonifacio et al., 2022; Yuan et al., 2021; Liu et al., 2022) and helping humans critique text (Saunders ∗Equal contribution. 1https://github.com/INK-USC/ RationaleHumanUtility Figure 1: An illustration of Human Utility of ratio- nales: Here, we show Chains of Thought (rationales) generated by GPT-3 in two scenarios. The first one is providing knowledge to the human to be able to answer the question, but the second rationale is not useful, and is in fact, misleading the human to answer incorrectly. et al., 2022). However, the opaque reasoning pro- cesses of these LMs pose serious concerns about their role in high-stakes decision-making (Bender et al., 2021; Doshi-Velez and Kim, 2017). Recently, many works have explored using LMs to generate fluent, human-like free-text rationales2 via natural language (Ehsan et al., 2018; Rajani et al., 2019a) that can explain their decisions. Further, rationales can reference things beyond the task input, and also support high flexibility in content, style, and length (Narang et al., …

Content truncated to 500 words