Evaluating Textual Representations through Image Generation

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 30–39 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 30 Evaluating Textual Representations through Image Generation Graham Spinks Department of Computer Science KU Leuven, Belgium graham.spinks@cs.kuleuven.be Marie-Francine Moens Department of Computer Science KU Leuven, Belgium sien.moens@cs.kuleuven.be Abstract We present a methodology for determining the quality of textual representations through the ability to generate images from them. Contin- uous representations of textual input are ubiq- uitous in modern Natural Language Process- ing techniques either at the core of machine learning algorithms or as the by-product at any given layer of a neural network. While current techniques to evaluate such represen- tations focus on their performance on partic- ular tasks, they don’t provide a clear under- standing of the level of informational detail that is stored within them, especially their abil- ity to represent spatial information. The cen- tral premise of this paper is that visual inspec- tion or analysis is the most convenient method to quickly and accurately determine informa- tion content. Through the use of text-to-image neural networks, we propose a new technique to compare the quality of textual representa- tions by visualizing their information content. The method is illustrated on a medical dataset where the correct representation of spatial in- formation and shorthands are of particular im- portance. For four different well-known tex- tual representations, we show with a quanti- tative analysis that some representations are consistently able to deliver higher quality vi- sualizations of the information content. Addi- tionally, we show that the quantitative analy- sis technique correlates with the judgment of a human expert evaluator in terms of alignment. 1 Introduction In this paper, a method is proposed to evaluate the quality of a textual representation by conditioning an image generation network on it. Neural networks implicitly construct represen- tations of a textual input by learning which fea- tures are important for the task at hand. It is not immediately possible however to assess the level of detail and structure that is retained in such a rep- resentation. Many systems often complement or replace the input with pre-trained representations that have the advantage of being constructed with a larger unlabeled corpus. Depending on the task, this practice sometimes significantly improves the performance of the network (Turian et al., 2010). On the one hand, this is due to the use of a larger unlabeled corpus which reduces data sparsity and thus improves generalization accuracy. On the other hand, representations often contain higher- level features that are fundamental for the task they are trained for. A neural network in a sep- arate task can thus rely on those features without having to discover them all over again. As the field of Natural Language Processing ad- vances and machine learning models expand to include multimodal information, the importance of understanding the level of detail and informa- tion that is retained in a textual representation only grows. Obtained representations can be employed in additional …

Content truncated to 500 words