Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5495–5510, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5495 Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements Yang Li∗, Gang Li∗, Luheng He, Jingjie Zheng, Hong Li†, Zhiwei Guan Google Research, Mountain View, CA 94043, USA Georgia Tech, Atlanta, GA 30332, USA† {liyang, leebird, luheng, jingjiezheng, zguan}@google.com Abstract Natural language descriptions of user interface (UI) elements such as alternative text are cru- cial for accessibility and language-based inter- action in general. Yet, these descriptions are constantly missing in mobile UIs. We pro- pose widget captioning, a novel task for auto- matically generating language descriptions for UI elements from multimodal input including both the image and the structural representa- tions of user interfaces. We collected a large- scale dataset for widget captioning with crowd- sourcing. Our dataset contains 162,859 lan- guage phrases created by human workers for annotating 61,285 UI elements across 21,750 unique UI screens. We thoroughly analyze the dataset, and train and evaluate a set of deep model configurations to investigate how each feature modality as well as the choice of learn- ing strategies impact the quality of predicted captions. The task formulation and the dataset as well as our benchmark models contribute a solid basis for this novel multimodal caption- ing task that connects language and user inter- faces. 1 Introduction Mobile apps come with a rich and diverse set of design styles, which are often more graphical and unconventional compared to traditional desktop ap- plications. Language descriptions of user interface (UI) elements—that we refer to as widget captions— are a precondition for many aspects of mobile UI usability. For example, accessibility services such as screen readers, e.g., Talkback (2019), rely on widget captions to make UI elements accessible to visually impaired users via text-to-speech technolo- gies. Importantly, widget captions are an enabler for many language-based interaction capabilities ∗ Equal contribution † Participated in the project during an internship at Google Research. View Hierarchy Structures play song add a new playlist Model Widget Captions Element Pixels Mobile UI Figure 1: Widget captioning is a task to generate lan- guage descriptions for UI elements that miss captions, given multimodal input of UI structures and screenshot images. These captions are crucial for accessibility and language-based interaction in general. such as voice commands and general screen under- standing efforts. However, a significant portion of mobile apps today lack widget captions in their user interfaces, which have stood out as a primary issue for mo- bile accessibility (Ross et al., 2018, 2017). More than half of image-based elements have missing captions (Ross et al., 2018). Beyond image-based ones, our analysis of a UI corpus here showed that a wide range of elements have missing captions. Existing tools for examining and fixing missing captions (AccessibilityScanner, 2019; AndroidLint, 2019; Zhang et al., 2018, 2017; Choo et al., 2019) require developers to manually compose a language description for each element, which imposes a sub- stantial overhead on developers. …

Content truncated to 500 words