Generating Animations from Screenplays

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 292–307 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 292 Generating Animations from Screenplays Yeyao Zhang1,3, Eleftheria Tsipidi1 , Sasha Schriber1, Mubbasir Kapadia1,2, Markus Gross1,3, Ashutosh Modi1 1Disney Research, 2Rutgers University, 3ETH Zurich yezhang@inf.ethz.ch {etsipidi,sasha.schriber}@disneyresearch.com mubbasir.kapadia@rutgers.edu {gross,ashutosh.modi}@disneyresearch.com Abstract Automatically generating animation from na- tural language text finds application in a num- ber of areas e.g. movie script writing, ins- tructional videos, and public safety. However, translating natural language text into anima- tion is a challenging task. Existing text-to- animation systems can handle only very sim- ple sentences, which limits their applications. In this paper, we develop a text-to-animation system which is capable of handling complex sentences. We achieve this by introducing a text simplification step into the process. Buil- ding on an existing animation generation sys- tem for screenwriting, we create a robust NLP pipeline to extract information from screen- plays and map them to the system’s knowledge base. We develop a set of linguistic transfor- mation rules that simplify complex sentences. Information extracted from the simplified sen- tences is used to generate a rough storyboard and video depicting the text. Our sentence sim- plification module outperforms existing sys- tems in terms of BLEU and SARI metrics.We further evaluated our system via a user study: 68 % participants believe that our system gene- rates reasonable animation from input screen- plays. 1. Introduction Generating animation from texts can be useful in many contexts e.g. movie script writing (Ma and Kevitt, 2006; Liu and Leung, 2006; Hanser et al., 2010), instructional videos (Lu and Zhang, 2002), and public safety (Johansson et al., 2004). Text-to- animation systems can be particularly valuable for screenwriting by enabling faster iteration, prototy- ping and proof of concept for content creators. In this paper, we propose a text-to-animation generation system. Given an input text describing a certain activity, the system generates a rough ani- mation of the text. We are addressing a practical setting, where we do not have any annotated data for training a supervised end-to-end system. The aim is not to generate a polished, final animation, but a pre-visualization of the input text. The pur- pose of the system is not to replace writers and artists, but to make their work more efficient and less tedious. We are aiming for a system which is robust and could be deployed in a production en- vironment. Existing text-to-animation systems for screenw- riting (§2) visualize stories by using a pipeline of Natural Language Processing (NLP) techniques for extracting information from texts and mapping them to appropriate action units in the animation engine. The NLP modules in these systems trans- late the input text into predefined intermediate ac- tion representations and the animation generation engine produces simple animation from these re- presentations. Although these systems can generate anima- tion from carefully handcrafted simple sentences, translating real screenplays into coherent anima- tion still remains a challenge. This can be attri- buted to the limitations of the NLP …

Content truncated to 500 words