A Low-Rank Approximation Approach to Learning Joint Embeddings of News Stories and Images for Timeline Summarization

Proceedings of NAACL-HLT 2016, pages 58–68, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics A Low-Rank Approximation Approach to Learning Joint Embeddings of News Stories and Images for Timeline Summarization William Yang Wang1∗, Yashar Mehdad3, Dragomir R. Radev2, Amanda Stent4 1School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213, USA 2Department of EECS, University of Michigan, Ann Arbor, MI 48109, USA 3Yahoo, Sunnyvale, CA 94089, USA and 4New York, NY 10036, USA yww@cs.cmu.edu, {ymehdad,stent}@yahoo-inc.com, radev@umich.edu Abstract A key challenge for timeline summarization is to generate a concise, yet complete storyline from large collections of news stories. Pre- vious studies in extractive timeline generation are limited in two ways: first, most prior work focuses on fully-observable ranking models or clustering models with hand-designed features that may not generalize well. Second, most summarization corpora are text-only, which means that text is the sole source of infor- mation considered in timeline summarization, and thus, the rich visual content from news images is ignored. To solve these issues, we leverage the success of matrix factorization techniques from recommender systems, and cast the problem as a sentence recommenda- tion task, using a representation learning ap- proach. To augment text-only corpora, for each candidate sentence in a news article, we take advantage of top-ranked relevant images from the Web and model the image using a convolutional neural network architecture. Fi- nally, we propose a scalable low-rank approx- imation approach for learning joint embed- dings of news stories and images. In experi- ments, we compare our model to various com- petitive baselines, and demonstrate the state- of-the-art performance of the proposed text- based and multimodal approaches. 1 Introduction Timeline summarization is the task of organizing crucial milestones of a news story in a temporal or- der, e.g. (Kedzie et al., 2014; Lin et al., 2012). A ∗This work was performed when William Wang and Dragomir Radev were visiting Yahoo NYC. timeline example for the 2010 British Oil spill gen- erated by our system is shown in Figure 1. The task is challenging, because the input often includes a large number of news articles as the story is de- veloping each day, but only a small portion of the key information is needed for timeline generation. In addition to the conciseness requirement, timeline summarization also has to be complete—all key in- formation, in whatever form, must be presented in the final summary. To distill key insights from news reports, prior work in summarization often relies on feature en- gineering, and uses clustering techniques (Radev et al., 2004b) to select important events to be included in the final summary. While this approach is un- supervised, the process of feature engineering is al- ways expensive, and the number of clusters is not easy to estimate. To present a complete summary, researchers from the natural language processing (NLP) community often solely rely on the textual information, while studies in the computer vision (CV) community rely solely on the image and video information. However, even though news images …

Content truncated to 500 words