Towards Better Semantic Understanding of Mobile Interfaces

Proceedings of the 29th International Conference on Computational Linguistics, pages 5636–5650 October 12–17, 2022. 5636 Towards Better Semantic Understanding of Mobile Interfaces Srinivas Sunkara1∗ Maria Wang1∗ Lijuan Liu1 Gilles Baechler1 Yu-Chung Hsiao1 Jindong (JD) Chen1 Abhanshu Sharma1 James Stout2 1Google Research, 2Google Abstract Improving the accessibility and automation ca- pabilities of mobile devices can have a signifi- cant positive impact on the daily lives of count- less users. To stimulate research in this di- rection, we release a human-annotated dataset with approximately 500k unique annotations aimed at increasing the understanding of the functionality of UI elements. This dataset augments images and view hierarchies from RICO, a large dataset of mobile UIs, with annotations for icons based on their shapes and semantics, and associations between dif- ferent elements and their corresponding text la- bels, resulting in a significant increase in the number of UI elements and the categories as- signed to them. We also release models us- ing image-only and multimodal inputs; we ex- periment with various architectures and study the benefits of using multimodal inputs on the new dataset. Our models demonstrate strong performance on an evaluation set of unseen apps, indicating their generalizability to newer screens. These models, combined with the new dataset, can enable innovative functional- ities like referring to UI elements by their la- bels, improved coverage and better semantics for icons etc., which would go a long way in making UIs more usable for everyone. 1 Introduction Mobile devices like phones and tablets have be- come ubiquitous and indispensable to carry out our daily activities. It is not an exaggeration to say that usage of mobile devices is becoming a re- quirement for full participation in society. Recent reports from the WHO and others (Organization, 2021; Peter Ackland and Bourne, 2017) estimate that around 2.2 billion people across the world have some form of vision impairment, out of which 36 million people are blind. Accessibility of mobile *Equal contribution, correspondence: {srini- vasksun,mariawang}@google.com devices is necessary for these visually impaired users to carry out their daily tasks and is an impor- tant tool for their social integration (Ladner, 2015). Accessibility of mobile apps has improved sig- nificantly over the past few years aided by develop- ments on two main fronts: Firstly, the development of screen readers, like VoiceOver (Apple, 2021c) on iOS and TalkBack (Accessibility, 2021e) on Android, enable visually impaired users to con- trol their phone in an eyes-free manner. Secondly, development tools and standards to enhance ac- cessibility, such as the Accessibility guidelines for iOS and Android (Accessibility, 2021a; Apple, 2021b), Android Accessibility Scanner (Accessibil- ity, 2021b), and iOS Accessibility Inspector (Apple, 2021a), have helped developers identify and fix ac- cessibility issues for applications. Among most of these utilities, the main source of accessibility data is the accessibility labels (Accessibility, 2021d) provided by app developers. These labels are speci- fied as attributes on a structured representation such as View Hierarchy, for the different UI elements on the screen and are available to screen readers (Accessibility, …

Content truncated to 500 words