Grounding Language with Points and Paths in Continuous Spaces

Proceedings of the Eighteenth Conference on Computational Language Learning, pages 58–67, Baltimore, Maryland USA, June 26-27 2014. c⃝2014 Association for Computational Linguistics Grounding Language with Points and Paths in Continuous Spaces Jacob Andreas and Dan Klein Computer Science Division University of California, Berkeley {jda,klein}@cs.berkeley.edu Abstract We present a model for generating path- valued interpretations of natural language text. Our model encodes a map from natural language descriptions to paths, mediated by segmentation variables which break the language into a discrete set of events, and alignment variables which reorder those events. Within an event, lexical weights capture the contribution of each word to the aligned path segment. We demonstrate the applicability of our model on three diverse tasks: a new color description task, a new financial news task and an established direction-following task. On all three, the model outperforms strong baselines, and on a hard variant of the direction-following task it achieves results close to the state-of-the-art system described in Vogel and Jurafsky (2010). 1 Introduction This paper introduces a probabilistic model for predicting grounded, real-valued trajectories from natural language text. A long tradition of re- search in compositional semantics has focused on discrete representations of meaning. The origi- nal focus of such work was on logical translation: mapping statements of natural language to a for- mal language like first-order logic (Zettlemoyer and Collins, 2005) or database queries (Zelle and Mooney, 1996). Subsequent work has integrated this logical translation with interpretation against a symbolic database (Liang et al., 2013). There has been a recent increase in interest in perceptual grounding, where lexical semantics anchor in perceptual variables (points, distances, etc.) derived from images or video. Bruni et al. (2014) describe a procedure for constructing word representations using text- and image-based dis- % Change 0.98 0.99 1.00 1.01 1.02 Hour of day 10 12 14 10 12 14 U.S. stocks rebound after bruising two-day swoon Figure 1: Example stock data. The chart displays index value over a two-day period (divided by the dotted line), while the accompanying headline de- scribes the observed behavior. tributional information. Yu and Siskind (2013) describe a model for identifying scenes given de- scriptions, and Golland et al. (2010), Kollar et al. (2010), and Krishnamurthy and Kollar (2013) de- scribe models for identifying individual compo- nents of scenes described by text. These all have the form of matching problems between text and observed groundings—what has been missing so far is the ability to generate grounded interpreta- tions from scratch, given only text. Our work continues in the tradition of this per- ceptual grounding work, but makes two contribu- tions. First, our approach is able to predict simple world states (and their evolution): for a general class of continuous domains, we produce a repre- sentation of p(world | text) that admits easy sam- pling and maximization. This makes it possible to produce grounded interpretations of text without reference to a pre-existing scene. Simultaneously, we extend the range of temporal phenomena that can be modeled—unlike the aforementioned spa- tial semantics …

Content truncated to 500 words