Easy as ABC? Facilitating Pictorial Communication via Semantically Enhanced Layout

CoNLL 2008: Proceedings of the 12th Conference on Computational Natural Language Learning, pages 119–126 Manchester, August 2008 Easy as ABC? Facilitating Pictorial Communication via Semantically Enhanced Layout Andrew B. Goldberg, Xiaojin Zhu, Charles R. Dyer, Mohamed Eldawy, Lijie Heng Department of Computer Sciences University of Wisconsin, Madison, WI 53706, USA {goldberg, jerryzhu, dyer, eldawy, ljheng}@cs.wisc.edu Abstract Pictorial communication systems convert natural language text into pictures to as- sist people with limited literacy. We define a novel and challenging problem: picture layout optimization. Given an input sen- tence, we seek the optimal way to lay out word icons such that the resulting picture best conveys the meaning of the input sen- tence. To this end, we propose a family of intuitive “ABC” layouts, which organize icons in three groups. We formalize layout optimization as a sequence labeling prob- lem, employing conditional random fields as our machine learning method. Enabled by novel applications of semantic role la- beling and syntactic parsing, our trained model makes layout predictions that agree well with human annotators. In addition, we conduct a user study to compare our ABC layout versus the standard linear lay- out. The study shows that our semantically enhanced layout is preferred by non-native speakers, suggesting it has the potential to be useful for people with other forms of limited literacy, too. 1 Introduction A picture is worth a thousand words—especially when you are someone with communicative dis- orders, a foreign language speaker, or a young child. Pictorial communication systems aim to au- tomatically convert general natural language text into meaningful pictures. A perfect pictorial c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. communication system can turn signs and opera- tion instructions into easy-to-understand graphical forms; combined with optical character recogni- tion input, a personal assistant device could create such visual translations on-the-fly without the help of a caretaker. Pictorial communication may also facilitate literacy development and rapid browsing of documents through pictorial summaries. Pictorial communication research is in its in- fancy with a spectrum of experimental systems, which we review in Section 2. At one end of the spectrum, some systems render highly realis- tic 3D scenes but require specific scene-descriptive language. At the other end, some systems per- form dictionary-based iconic transliteration (turn- ing words into icons1 one by one) on arbitrary text but the pictures can be hard to understand. We are interested in using pictorial communication as an assistive communication tool. Thus, our system needs to be able to handle general text yet produce easy-to-understand pictures, which is in the middle of the spectrum. To this end, our system adopts a “collage” approach (Zhu et al., 2007). Given a piece of text (e.g., a sentence), it first identifies im- portant and easy-to-depict words (or phrases) with natural language processing (NLP) techniques. It then finds one good icon per word, either from a manually created picture-dictionary, or via image analysis on image search results. Finally, it lays out the icons to …

Content truncated to 500 words