TuringAdvice: A Generative and Dynamic Evaluation of Language Use

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4856–4880 June 6–11, 2021. ©2021 Association for Computational Linguistics 4856 TuringAdvice: A Generative and Dynamic Evaluation of Language Use Rowan Zellers♠ Ari Holtzman♠ Elizabeth Clark♠ Lianhui Qin♠ Ali Farhadi♠ Yejin Choi♠♥ ♠Paul G. Allen School of Computer Science & Engineering, University of Washington ♥Allen Institute for Artificial Intelligence rowanzellers.com/advice Abstract We propose TuringAdvice, a new challenge task and dataset for language understanding models. Given a written situation that a real person is currently facing, a model must gen- erate helpful advice in natural language. Our evaluation framework tests a fundamental as- pect of human language understanding: our ability to use language to resolve open-ended situations by communicating with each other. Empirical results show that today’s models struggle at TuringAdvice, even multibillion pa- rameter models finetuned on 600k in-domain training examples. The best model, a finetuned T5, writes advice that is at least as helpful as human-written advice in only 14% of cases; a much larger non-finetunable GPT3 model does even worse at 4%. This low performance re- veals language understanding errors that are hard to spot outside of a generative setting, showing much room for progress. 1 Introduction Language models today are getting ever-larger, and are being trained on ever-increasing quantities of text. For an immense compute cost, these models like T5 (Raffel et al., 2019) and GPT3 (Brown et al., 2020) show gains on a variety of standard NLP benchmarks – often even outperforming humans. Yet, when a giant model like T5 generates lan- guage, we observe clear gaps between machine- level and human-level language understanding – even after it has been finetuned for the task at hand. Consider Figure 1, in which a woman asks for advice. She is assigned to dissect an animal for her class project, but has extreme anxiety about dead animals – and her teacher refused to give her another assignment. Humans can respond with helpful advice, reflecting our unique ability of real- world language use: to communicate and tackle open-ended issues. The helpful advice in this ex- T5 I’d send a short email to the next higher-up authority figure, ideally a counselor. Be forthright; it’s the best approach when self- advocating as a student. Go to your teacher and say "I'm asking you to do a project that requires me to see dead animals. This is a dealbreaker." If she doesn’t concede, tell your principal about your trauma. I have to do a dissection for my high school class, but I’m distressed by dead animals. Last time we dissected an animal in class, I had a panic attack. I asked my teacher for another assignment, but she refused. I don't want to play a 'victim' card, but I don't know what to do. Help! Helpful Not helpful Figure 1: TuringAdvice. Humans are natural experts at using language to successfully address situations that arise, such as giving advice. We introduce a new frame- work, …

Content truncated to 500 words