MOTIF: Contextualized Images for Complex Words to Improve Human Reading

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2468–2477 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2468 MOTIF: Contextualized Images for Complex Words to Improve Human Reading Xintong Wang∗1, Florian Schneider∗1, ¨Ozge Alac¸am2, Prateek Chaudhury3, Chris Biemann1 1Universit¨at Hamburg, 2Universit¨at Bielefeld, 3Indian Institute of Technology Delhi, ∗Equal Contribution {xintong.wang, florian.schneider-1, christian.biemann}@uni-hamburg.de oezge.alacam@uni-bielefeld.de, prateekchaudhury@gmail.com Abstract MOTIF (MultimOdal ConTextualized Images For Language Learners) is a multimodal dataset that consists of 1125 com- prehension texts retrieved from Wikipedia Simple Corpus. Allowing multimodal processing or enriching the context with multimodal information has proven imperative for many learning tasks, specifically for second language (L2) learning. In this respect, several traditional NLP approaches can assist L2 readers in text comprehension processes, such as simplifying text or giving dictionary descriptions for complex words. As nicely stated in the well-known proverb, sometimes “a picture is worth a thousand words” and an image can successfully complement the verbal message by enriching the representation, like in Pictionary books. This multimodal support can also assist on-the-fly text reading experience by providing a multimodal tool that chooses and displays the most relevant images for the complex words, given the text context. This study mainly focuses on one of the key components to achieving this goal; collecting a multimodal dataset enriched with complex word annotation and validated image match. Keywords: Context-dependent image retrieval, L2 reading material, Complex word identification 1. Introduction Whether in human cognitive processes or computa- tional systems, multimodal information is crucial for adequate concept formation, accordingly for language acquisition. Babies learn their native language by com- bining words with visual cues, e.g., the sound of the word “cat”, an image of a cat, and a cat sound are all essential for the concept of “cat”. Over the past two decades, literature has provided convincing evidence on the facilitating role of cross-modal information in language acquisition (Ecalle et al., 2009; Dalton and Grisham, 2011; Hahn et al., 2014; Gerbier et al., 2018; Xie et al., 2019; Albahiri and Alhaj, 2020). Although words are powerful symbolic representa- tions, explaining the message (communicative intent) verbally yields unwieldy over specified sentences. Suc- cessful communication in daily communication set- tings usually involves linguistic information accompa- nied by other modalities like visual representations, gestures, or audio. The advantage of multimodal in- formation holds for second language (L2) acquisition. Modern language learning applications or dictionaries like Babbel1 or Duolingo2 benefit from multimodality by using audio, visual illustrations, and video to en- hance the L2 learning experience. There are several approaches to assist non-native speakers in their reading activities. Through Lexical Simplification (LS), complex words can be replaced with simpler alternatives while preserving the mean- ing and syntactic function. It has been shown that LS leads to better text comprehension, improving text re- call, especially for L2 learners at lower proficiency lev- 1https://www.babbel.com/ 2https://www.duolingo.com/ els (Rets and Rogaten, 2021). Instead of automatically simplifying the text, another approach would be to provide additional information about the complex word/phrase. …

Content truncated to 500 words