Semantic Relatedness Based Re-ranker for Text Spotting

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3451–3457, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3451 Semantic Relatedness Based Re-ranker for Text Spotting Ahmed Sabir1 Francesc Moreno-Noguer2 Llu´ıs Padr´o1 1 Universitat Polit`ecnica de Catalunya, TALP Research Center, Barcelona, Spain 2 Institut de Rob`otica i Inform`atica Industrial, CSIC-UPC, Barcelona, Spain asabir@cs.upc.edu, fmoreno@iri.upc.edu, padro@cs.upc.edu Abstract Applications such as textual entailment, pla- giarism detection or document clustering rely on the notion of semantic similarity, and are usually approached with dimension reduction techniques like LDA or with embedding-based neural approaches. We present a scenario where semantic similarity is not enough, and we devise a neural approach to learn seman- tic relatedness. The scenario is text spotting in the wild, where a text in an image (e.g. street sign, advertisement or bus destination) must be identified and recognized. Our goal is to improve the performance of vision systems by leveraging semantic information. Our ratio- nale is that the text to be spotted is often re- lated to the image context in which it appears (word pairs such as Delta–airplane, or quar- ters–parking are not similar, but are clearly re- lated). We show how learning a word-to-word or word-to-sentence relatedness score can im- prove the performance of text spotting systems up to 2.9 points, outperforming other measures in a benchmark dataset. 1 Introduction Deep learning has been successful in tasks related to deciding whether two short pieces of text refer to the same topic, e.g. semantic textual similarity (Cer et al., 2018), textual entailment (Parikh et al., 2016) or answer ranking for Q&A (Severyn and Moschitti, 2015). However, other tasks require a broader perspec- tive to decide whether two text fragments are re- lated more than whether they are similar. In this work, we describe one of such tasks, and we re- train some of the existing sentence similarity ap- proaches to learn this semantic relatedness. We also present a new Deep Neural Network (DNN) that outperforms existing approaches when ap- plied to this particular scenario. The task we tackle is Text Spotting, which is the problem of recognizing text that appears in unre- stricted images (a.k.a. text in the wild) such as traffic signs, commercial ads, or shop names. Cur- rent state-of-the-art results on this task are far from those of OCR systems with simple backgrounds. Existing approaches to Text Spotting usually di- vide the problem in two fundamental tasks: 1) text detection, consisting of selecting the image regions likely to contain texts, and 2) text recogni- tion, that converts the images within these bound- ing boxes into a readable string. In this work, we focus on the recognition stage, aiming to prove that semantic relatedness between the image con- text and the recognized text can be useful to boost the system performance. We use existing pre- trained architectures for Text Recognition, and add a shallow deep-network that performs a post- processing operation to re-rank …

Content truncated to 500 words