Neural Naturalist: Generating Fine-Grained Image Comparisons

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 708–717, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 708 Neural Naturalist: Generating Fine-Grained Image Comparisons Maxwell Forbes 🦉 Christine Kaeser-Chen 🦜 Piyush Sharma 🦜 Serge Belongie 🦜 🕊 🦉 University of Washington 🦜 Google Research 🕊 Cornell University and Cornell Tech mbforbes@cs.washington.edu {christinech,piyushsharma}@google.com sjb344@cornell.edu https://mbforbes.github.io/neural-naturalist/ Abstract We introduce the new Birds-to-Words dataset of 41k sentences describing fine-grained dif- ferences between photographs of birds. The language collected is highly detailed, while remaining understandable to the everyday observer (e.g., “heart-shaped face,” “squat body”). Paragraph-length descriptions natu- rally adapt to varying levels of taxonomic and visual distance—drawn from a novel strati- fied sampling approach—with the appropriate level of detail. We propose a new model called Neural Naturalist that uses a joint image en- coding and comparative module to generate comparative language, and evaluate the results with humans who must use the descriptions to distinguish real images. Our results indicate promising potential for neural models to explain differences in visual embedding space using natural language, as well as a concrete path for machine learning to aid citizen scientists in their effort to preserve biodiversity. 1 Introduction Humans are adept at making fine-grained compar- isons, but sometimes require aid in distinguishing visually similar classes. Take, for example, a cit- izen science effort like iNaturalist,1 where every- day people photograph wildlife, and the commu- nity reaches a consensus on the taxonomic label for each instance. Many species are visually sim- ilar (e.g., Figure 1, top), making them difficult for a casual observer to label correctly. This puts an undue strain on lieutenants of the citizen science community to curate and justify labels for a large number of instances. While everyone may be ca- pable of making such distinctions visually, non- experts require training to know what to look for. 🦉 Work done during an internship at Google. 1https://www.inaturalist.org “Animal 2 looks smaller and has a stouter, darker bill than Animal 1. Animal 2 has black spots on its wings. Animal 2 has a black hood that extends down onto its breast, and the rest of its breast is white with orange only on its sides. In comparison, Animal 1’s breast is entirely orange.” “Animal 2 is brightly red-colored all over, except for a black oval around its beak. Animal 1 has more muted red and grey colors.” vs perceptual difficulty: medium highly detailed fewer details body part descriptive phrase perceptual difficulty: high Animal 1 Animal 1 Animal 1 Animal 1 Animal 2 Animal 2 Animal 2 Animal 2 vs Figure 1: The Birds-to-Words dataset: comparative de- scriptions adapt naturally to the appropriate level of de- tail (orange underlines). A difficult distinction (TOP) is given a longer and more fined-grained comparison than an easier one (BOTTOM). Annotators organically use everyday language to refer to parts (green highlights). Field guides exist for the purpose helping peo- ple learn how to distinguish …

Content truncated to 500 words