Worldly Wise (WoW) - Cross-Lingual Knowledge Fusion for Fact-based Visual Spoken-Question Answering

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1908–1919 June 6–11, 2021. ©2021 Association for Computational Linguistics 1908 Worldly Wise (WoW) - Cross-Lingual Knowledge Fusion for Fact-based Visual Spoken-Question Answering Kiran Ramnath1, Leda Sari1, Mark Hasegawa-Johnson1, and Chang Yoo2 1University of Illinois, Urbana-Champaign 2KAIST 1{kiranr2, lsari2, jhasegaw}@illinois.edu 2{cd_yoo}@kaist.ac.kr Abstract Although Question-Answering has long been of research interest, its accessibility to users through a speech interface and its support to multiple languages have not been addressed in prior studies. Towards these ends, we present a new task and a synthetically-generated dataset to do Fact-based Visual Spoken-Question An- swering (FVSQA). FVSQA is based on the FVQA dataset, which requires a system to re- trieve an entity from Knowledge Graphs (KGs) to answer a question about an image. In FVSQA, the question is spoken rather than typed. Three sub-tasks are proposed: (1) speech-to-text based, (2) end-to-end, without speech-to-text as an intermediate component, and (3) cross-lingual, in which the question is spoken in a language different from that in which the KG is recorded. The end-to-end and cross-lingual tasks are the first to require world knowledge from a multi-relational KG as a dif- ferentiable layer in an end-to-end spoken lan- guage understanding task, hence the proposed reference implementation is called Worldly- Wise (WoW). WoW is shown to perform end- to-end cross-lingual FVSQA at same levels of accuracy across 3 languages - English, Hindi, and Turkish. 1 Introduction Imagine being able to ask your voice assistant a question in any language, to learn some trivia about your favorite movie star. This task falls in the realm of Knowledge-based Question Answering (QA). One such challenging QA task is that of Fact-based Visual Question Answering (FVQA) (Wang et al., 2018) which seeks to imitate how humans lever- age background common-sense knowledge when answering visual questions. This task ensures that answering each question about an image requires external knowledge not directly available within the image or the text of the question. (see Fig. 1). The external information is provided in the form of knowledge graphs, which are multi-relational Figure 1: Example of a fact-based visual question Question - Which object in this image can be found in a Jazz Club? Supporting fact - You are likely to find [[a trumpet]] in [[a jazz club]] Subject, Predicate, Object - (Trumpet, AtLocation, Jazz Club) Answer - Trumpet graphs, storing relational representations between entities. The entities could be single words or phrases of words that denote objects or concepts. Such tasks, though widely studied, exist mostly for well-resourced languages (Goyal et al., 2017; Wang et al., 2018). These languages generally also have mature Automatic Speech Recognition (ASR) systems and language models. The accompanying Knowledge Graphs (KGs) also tend to be limited to languages that are well-resourced (Auer et al., 2007; Tandon et al., 2014; Liu and Singh, 2004). Against this background, it is worthwhile to think of building end-to-end systems which directly use speech signals as input, that can …

Content truncated to 500 words