Benchmarking Hierarchical Script Knowledge

Proceedings of NAACL-HLT 2019, pages 4077–4085 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4077 Benchmarking Hierarchical Script Knowledge Yonatan Bisk1 Jan Buys1 Karl Pichotta∗ Yejin Choi1,2 1Paul G. Allen School of Computer Science and Engineering, University of Washington 2Allen Institute for Artificial Intelligence {ybisk, jbuys}@cs.washington.edu Abstract Understanding procedural language requires reasoning about both hierarchical and tempo- ral relations between events. For example, “boiling pasta” is a sub-event of “making a pasta dish”, typically happens before “draining pasta,” and requires the use of omitted tools (e.g. a strainer, sink...). While people are able to choose when and how to use abstract ver- sus concrete instructions, the NLP community lacks corpora and tasks for evaluating if our models can do the same. In this paper, we in- troduce KIDSCOOK, a parallel script corpus, as well as a cloze task which matches video captions with missing procedural details. Ex- perimental results show that state-of-the-art models struggle at this task, which requires inducing functional commonsense knowledge not explicitly stated in text. 1 Introduction The level of detail used in natural language com- munication varies: descriptive or instructive text for experts may elide over details the reader can seamlessly infer, while text for more novice audi- ences may be more verbose. A given document typically adheres to a single level of verbosity suited to its presumed audience (Grice, 1975), so learning correspondences between abstract and detailed descriptions of similar concepts from text is a challenging problem. Commonsense knowledge of how complex events decompose into stereotypical sequences of simpler events is a necessary component of a sys- tem that can automatically understand and rea- son about different types of discourse. Hierarchi- cal correspondences between abstract and detailed representations of concepts and events were an im- portant aspect of the original formulation of scripts for natural language understanding (Schank and ∗Author now at Google. Work done while unaffiliated. 1. Take the strainer with the pasta and pour the pasta into the sauce. 2. Stir the pasta into sauce while it is in the pan. 3. Let the pasta and sauce simmer for a few minutes. Add pasta to the sauce t2 <latexit sha1_base64="P4+gfywhwn2dG1TBnMWEbekCJE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im/GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW/4PCofHzSNnGqGW+xWM a6G1DpVC8hQIl7ya0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKDzioDcoVt+ouQNaJl5MK5GgOyl/9YczSiCtkhrT89wE/YxqFEzyWamfGp5QNqEj3rNU0YgbP1ucOiMXVhmSMNa2FJKF+nsio5Ex0yiwnRHFsVn15uJ/Xi/F8NrPhEpS5IotF4WpJBiT+d9kKDRnKeWUKaFvZWwMdWUoU2nZEPwVl9eJ+1a1XOr3v1VpXGTx1GEMziHS/CgDg24gya0gMEInuEV3hzpvDjvzseyteDkM6fwB87nDwcjZ0=</latexit> <latexit sha1_base64="P4+gfywhwn2dG1TBnMWEbekCJE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im/GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW/4PCofHzSNnGqGW+xWM a6G1DpVC8hQIl7ya0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKDzioDcoVt+ouQNaJl5MK5GgOyl/9YczSiCtkhrT89wE/YxqFEzyWamfGp5QNqEj3rNU0YgbP1ucOiMXVhmSMNa2FJKF+nsio5Ex0yiwnRHFsVn15uJ/Xi/F8NrPhEpS5IotF4WpJBiT+d9kKDRnKeWUKaFvZWwMdWUoU2nZEPwVl9eJ+1a1XOr3v1VpXGTx1GEMziHS/CgDg24gya0gMEInuEV3hzpvDjvzseyteDkM6fwB87nDwcjZ0=</latexit> <latexit sha1_base64="P4+gfywhwn2dG1TBnMWEbekCJE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im/GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW/4PCofHzSNnGqGW+xWM a6G1DpVC8hQIl7ya0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKDzioDcoVt+ouQNaJl5MK5GgOyl/9YczSiCtkhrT89wE/YxqFEzyWamfGp5QNqEj3rNU0YgbP1ucOiMXVhmSMNa2FJKF+nsio5Ex0yiwnRHFsVn15uJ/Xi/F8NrPhEpS5IotF4WpJBiT+d9kKDRnKeWUKaFvZWwMdWUoU2nZEPwVl9eJ+1a1XOr3v1VpXGTx1GEMziHS/CgDg24gya0gMEInuEV3hzpvDjvzseyteDkM6fwB87nDwcjZ0=</latexit> <latexit sha1_base64="P4+gfywhwn2dG1TBnMWEbekCJE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im/GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW/4PCofHzSNnGqGW+xWM a6G1DpVC8hQIl7ya0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKDzioDcoVt+ouQNaJl5MK5GgOyl/9YczSiCtkhrT89wE/YxqFEzyWamfGp5QNqEj3rNU0YgbP1ucOiMXVhmSMNa2FJKF+nsio5Ex0yiwnRHFsVn15uJ/Xi/F8NrPhEpS5IotF4WpJBiT+d9kKDRnKeWUKaFvZWwMdWUoU2nZEPwVl9eJ+1a1XOr3v1VpXGTx1GEMziHS/CgDg24gya0gMEInuEV3hzpvDjvzseyteDkM6fwB87nDwcjZ0=</latexit> 1. Put a large pot half full of water on the stove. 2. Turn the heat on under the pot and wait for the water to boil hard. 3. Pour the pasta into the boiling water. Cook the pasta t0 <latexit sha1_base64="apCl0Q/pUAKLQ3uBGKSWYytus=">AB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD4 9aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpAfte3616NW8Oskr8glShQKPvfvUGCctirpBJakzX91IMcqpRMmnlV5meErZmA51JFY26CfH7qlJxZUCiRNtSObq74mcxsZM4tB2xhRHZtmbif953Qyj6yAXKs2QK7ZYFGWSYEJmf5OB0JyhnFhCmRb2VsJGVFOGNp2KDcFfnmVtC5qvlfz7y+r9ZsijKcwCmcgw9XUIc7aEATGAzhGV7hzZHOi/PufCxaS04xcwx/4Hz+AQUjZs=</latexit> <latexit sha1_base64="apCl0Q/pUAKLQ3uBGKSWYytus=">AB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD4 9aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpAfte3616NW8Oskr8glShQKPvfvUGCctirpBJakzX91IMcqpRMmnlV5meErZmA51JFY26CfH7qlJxZUCiRNtSObq74mcxsZM4tB2xhRHZtmbif953Qyj6yAXKs2QK7ZYFGWSYEJmf5OB0JyhnFhCmRb2VsJGVFOGNp2KDcFfnmVtC5qvlfz7y+r9ZsijKcwCmcgw9XUIc7aEATGAzhGV7hzZHOi/PufCxaS04xcwx/4Hz+AQUjZs=</latexit> <latexit sha1_base64="apCl0Q/pUAKLQ3uBGKSWYytus=">AB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD4 9aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpAfte3616NW8Oskr8glShQKPvfvUGCctirpBJakzX91IMcqpRMmnlV5meErZmA51JFY26CfH7qlJxZUCiRNtSObq74mcxsZM4tB2xhRHZtmbif953Qyj6yAXKs2QK7ZYFGWSYEJmf5OB0JyhnFhCmRb2VsJGVFOGNp2KDcFfnmVtC5qvlfz7y+r9ZsijKcwCmcgw9XUIc7aEATGAzhGV7hzZHOi/PufCxaS04xcwx/4Hz+AQUjZs=</latexit> <latexit sha1_base64="apCl0Q/pUAKLQ3uBGKSWYytus=">AB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD4 9aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpAfte3616NW8Oskr8glShQKPvfvUGCctirpBJakzX91IMcqpRMmnlV5meErZmA51JFY26CfH7qlJxZUCiRNtSObq74mcxsZM4tB2xhRHZtmbif953Qyj6yAXKs2QK7ZYFGWSYEJmf5OB0JyhnFhCmRb2VsJGVFOGNp2KDcFfnmVtC5qvlfz7y+r9ZsijKcwCmcgw9XUIc7aEATGAzhGV7hzZHOi/PufCxaS04xcwx/4Hz+AQUjZs=</latexit> … … Drain the pasta 1. Put the strainer in the sink. 2. Once the pot with pasta is cool enough, grab it by the handles. 3. Pour the pasta and water into the strainer in the sink. 4. Pick up the strainer and shake it a little bit so more water comes out. t1 <latexit sha1_base64="FVaQdjUWZVLyUalbTR1NMDoKM=">AB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk 0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpAft+3616NW8Oskr8glShQKPvfvUGCctirpBJakzX91IMcqpRMmnlV5meErZmA51JFY26CfH7qlJxZUCiRNtSObq74mcxsZM4tB2xhRHZtmbif953Qyj6yAXKs2QK7ZYFGWSYEJmf5OB0JyhnFhCmRb2VsJGVFOGNp2KDcFfnmVtC5qvlfz7y+r9ZsijKcwCmcgw9XUIc7aEATGAzhGV7hzZHOi/PufCxaS04xcwx/4Hz+AWYjZw=</latexit> <latexit sha1_base64="FVaQdjUWZVLyUalbTR1NMDoKM=">AB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk 0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpAft+3616NW8Oskr8glShQKPvfvUGCctirpBJakzX91IMcqpRMmnlV5meErZmA51JFY26CfH7qlJxZUCiRNtSObq74mcxsZM4tB2xhRHZtmbif953Qyj6yAXKs2QK7ZYFGWSYEJmf5OB0JyhnFhCmRb2VsJGVFOGNp2KDcFfnmVtC5qvlfz7y+r9ZsijKcwCmcgw9XUIc7aEATGAzhGV7hzZHOi/PufCxaS04xcwx/4Hz+AWYjZw=</latexit> <latexit sha1_base64="FVaQdjUWZVLyUalbTR1NMDoKM=">AB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk 0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpAft+3616NW8Oskr8glShQKPvfvUGCctirpBJakzX91IMcqpRMmnlV5meErZmA51JFY26CfH7qlJxZUCiRNtSObq74mcxsZM4tB2xhRHZtmbif953Qyj6yAXKs2QK7ZYFGWSYEJmf5OB0JyhnFhCmRb2VsJGVFOGNp2KDcFfnmVtC5qvlfz7y+r9ZsijKcwCmcgw9XUIc7aEATGAzhGV7hzZHOi/PufCxaS04xcwx/4Hz+AWYjZw=</latexit> <latexit sha1_base64="FVaQdjUWZVLyUalbTR1NMDoKM=">AB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk 0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpAft+3616NW8Oskr8glShQKPvfvUGCctirpBJakzX91IMcqpRMmnlV5meErZmA51JFY26CfH7qlJxZUCiRNtSObq74mcxsZM4tB2xhRHZtmbif953Qyj6yAXKs2QK7ZYFGWSYEJmf5OB0JyhnFhCmRb2VsJGVFOGNp2KDcFfnmVtC5qvlfz7y+r9ZsijKcwCmcgw9XUIc7aEATGAzhGV7hzZHOi/PufCxaS04xcwx/4Hz+AWYjZw=</latexit> Figure 1: An example KIDSCOOK sequence with mul- tiple …

Content truncated to 500 words