Image Description Dataset for Language Learners

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6814–6821 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6814 Image Description Dataset for Language Learners Kento Tanaka1, Taichi Nishimura1, Hiroaki Nanjo2, Keisuke Shirai1, Hirotaka Kameko2, Masatake Dantsuji3 1: Graduate School of Informatics, Kyoto University, 2: Academic Center for Computing and Media Studies, Kyoto University 3: Faculty of English and Global Communication, Kyoto Tachibana University 1,2: Yoshidahonmachi, Sakyo-ku, Kyoto, Japan, 606-8501 3: 34 Oyake Yamada, Yamashina-ku, Kyoto, Japan, 607-8175 1tanaka.kento.s07@kyoto-u.jp, 1{nishimura.taichi43x,shirai.keisuke.64x}@st.kyoto-u.ac.jp 2nanjo@media.kyoto-u.ac.jp 2kameko@i.kyoto-u.ac.jp 3dantsuji@tachibana-u.ac.jp Abstract We focus on image description and a corresponding assessment system for language learners. To achieve automatic assessment of image description, we construct a novel dataset, the Language Learner Image Description (LLID) dataset, which consists of images, their descriptions, and assessment annotations. Then, we propose a novel task of automatic error correction for image description, and we develop a baseline model that encodes multimodal information from a learner sentence with an image and accurately decodes a corrected sentence. Our experimental results show that the developed model can revise errors that cannot be revised without an image. Keywords: Image Description, Sentence Error Correction, Language Learning 1. Introduction Language learning involves four basic skills: reading, listening, writing, and speaking. The former two are perception skills, while the latter two are production skills. For developing perception skills, learners are typically given texts or speech and asked to answer multiple-choice questions. Because learners’ answers are limited, automatic assessment is not difficult in this situation. In contrast, for developing production skills, learners produce various texts or speech, and an auto- matic assessment system must handle a diverse range of answers and/or reduce the answer diversity. Among possible approaches for reducing the answer diversity, one approach is to use a translation task. Given a text written in a learner’s native language, the learner is asked to produce the text in the target lan- guage. The range of produced texts is expected to be limited by the given texts. However, automatic assess- ment of the translated texts would require a system that depends on both the learner’s native language and the target language. Another possible approach is to use an image description task1, in which a learner is asked to produce a sentence that describes an image. This task also limits the learner’s output to some extent, and it is independent of the learner’s native language because the questions consist only of images. Accordingly, this approach would enable learners with different native languages to use the same automatic assessment sys- tem for a given target language. 1Note that this task is usually called “picture descrip- tion.” In the information processing field, however, the term “image” is used instead of “picture,” and we thus use the term “image description” instead of “picture description” in this paper. Figure 1: Concept of the assessment system for image description. Given the above background, in this paper, we focus on image description and a corresponding assessment system, as shown in …

Content truncated to 500 words