Query Generation for Multimodal Documents

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 659–668 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 659 Query Generation for Multimodal Documents Kyungho Kim1, Kyungjae Lee1, Seung-won Hwang2∗, Young-In Song3 and Seungwook Lee3 1Yonsei University, 2Seoul National University, 3NAVER 1{ggdg12, lkj0509}@yonsei.ac.kr, 2seungwon.hwang@gmail.com, 3{song.youngin, swook.lee}@navercorp.com Abstract This paper studies the problem of generating likely queries for multimodal documents with images. Our application scenario is enabling efficient “first-stage retrieval” of relevant doc- uments, by attaching generated queries to doc- uments before indexing. We can then index this expanded text to efficiently narrow down to candidate matches using inverted index, so that expensive reranking can follow. Our eval- uation results show that our proposed multi- modal representation meaningfully improves relevance ranking. More importantly, our framework can achieve the state of the art in the first-stage retrieval scenarios. 1 Introduction As more documents on the web are generated and consumed by mobile devices with cameras, docu- ments are often multimodal, containing informa- tion in both text and image modalities. This poses a new challenge of finding relevance documents across modalities. More formally, the relevance of document, consisting of text t and image i, to the given query keywords q, should be modeled as a trimodal function f(q, t, i), rather than a simple lexical match between q and t (or, BM25 baseline) assuming the semantics of image i is fully rep- resented by the surrounding text (or, paired-text assumption). Prior research observes that paired-text assump- tion is often violated (Henning and Ewerth, 2017) – for example, some semantics can be better cap- tured in image modality, and may not (or, can- not) be described in text. Meanwhile, BM25 base- line (Robertson et al., 1994) would fail to serve queries for such semantics. ∗correspond to seungwon.hwang@gmail.com To overcome this limitation of BM25, one may model relevance from query as a trimodal func- tion f(q, t, i) (Nian et al., 2017; Kordan and Kotov, 2018) instead, but they require runtime invocation of f for the given query q with all potential docu- ment matches. This incurs a prohibitive runtime overhead, unacceptable for search engines find- ing results online. A common practice is to use a cheap BM25 ranking as a “first-stage retrieval”, efficiently supported by inverted index, to quickly narrow down to a few candidate documents, then evaluate f(q, t, i). However, due to simple nature of BM25 using exact term matching, a document will be missed, if the query term is absent in t, even though it semantically matches image or another term in the text. Our contribution is to keep first-stage retrieval as efficient as BM25, but enable multimodal se- mantic matching, using Query Generation (QG) before indexing. More specifically, we generate a likely query q from a joint modeling of t and i, to create an expanded text t′ = q ∪ t such that (q, t′) pair has more lexical overlaps, or better paired than (q, t), for first-phase retrieval. Specifically, …

Content truncated to 500 words