Interpretable Neural Architectures for Attributing an Ad’s Performance to its Writing Style

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 125–135 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 125 Interpretable Neural Architectures for Attributing an Ad’s Performance to its Writing Style Reid Pryzant∗ Stanford University rpryzant@stanford.edu Kazoo Sone Google sone@google.com Sugato Basu Google sugato@google.com Abstract How much does “free shipping!” help an ad- vertisement’s ability to persuade? This paper presents two methods for performance attri- bution: finding the degree to which an out- come can be attributed to parts of a text while controlling for potential confounders1. Both algorithms are based on interpreting the be- haviors and parameters of trained neural net- works. One method uses a CNN to encode the text, an adversarial objective function to con- trol for confounders, and projects its weights onto its activations to interpret the importance of each phrase towards each output class. The other method leverages residualization to con- trol for confounds and performs interpreta- tion by aggregating over learned word vec- tors. We demonstrate these algorithms’ ef- ficacy on 118,000 internet search advertise- ments and outcomes, finding language indica- tive of high and low click through rate (CTR) regardless of who the ad is by or what it is for. Our results suggest the proposed algo- rithms are high performance and data efficient, able to glean actionable insights from fewer than 10,000 data points. We find that quick, easy, and authoritative language is associated with success, while lackluster embellishment is related to failure. These findings agree with the advertising industry’s emperical wisdom, automatically revealing insights which previ- ously required manual A/B testing to discover. 1 Introduction A text’s style can affect our cognitive re- sponses and attitudes, thereby influencing behav- ior (Spence, 1983; Van Laer et al., 2013). The pre- dictive relationship between language and behav- ior has been well studied in applications of NLP to ∗This work was conducted while the first author was do- ing internship at Google. 1Our code is available at github.com/rpryzant/ deconfounded_lexicon_induction/tree/ master/text-performance-attribution tasks like linking text to sales figures (Ho and Wu, 1999; Pryzant et al., 2017) and voter preference (Luntz, 2007; Ansolabehere and Iyengar, 1995). In this paper, we are interested in interpret- ing rather than predicting the relationship between language and behavior. We focus on a specific in- stance: the relationship between the way a search advertisement is written and internet user behav- ior as measured by click through rate (CTR). In this study CTR is the ratio of clicks to impres- sions over a 90-day period, i.e. the probability of a click, given the person saw the ad. Our goal is to develop a method for performance attribution in textual advertisements: identifying lexical fea- tures (words, phrases, etc.) to which we can at- tribute the success (or failure) of a search ad, re- gardless of who created the advertisement or what it is selling. Identifying linguistic features that are associ- ated with various outcomes is a common activity among machine learning scientists and practition- ers. …

Content truncated to 500 words