Summary: 25 large-scale experiments with over 2 million households in co-operation with a large retailer, a stockbroker, and Yahoo show that it is extremely difficult to identify advertising ROI. Why? Because there is such massive cross-sectional variation across households in spending; plus huge variation in spending by the same household across time; and huge variation in purchase timing (sometimes a household buys twice in two weeks, sometimes only once in a year or not at all) – this massive variation makes it almost impossible to distinguish advertising effects from random noise. This is the case even when one has individual household data that matches both ad exposure and purchasing.
The market context
Here is some of the detail of the market and advertising context. I have simplified & explained the detail of the spending that is in the original study, to make it clearer to readers of this post. The context is a retailer, that can target ads to specific households. Furthermore it knows the exact amount of money those hosueholds spend with it. The study also uses a stockbroking firm that advertises to consumers with more or less the same sort of figures as below.
Typical scenario: An advertiser is going to hit households with approximately 35 display ads in a 2-week campaign period. The advertised product has a gross margin of 50%. The cost of the advertising per household is 14 cents (based on a price of $4 per thousand people, per ad = .4 cents per ad x 35 = 14 cents) for the campaign.
The average sales per consumer is $7 in the time period, but the standard deviation (i.e. the variation in sales across people) is $75. This means households vary from $0 to hundreds of dollars in purchases over the expected duration of the campaign.
Next, the advertising ROI goal is a 25% ROI. Spending 14 cents per person and getting a 25% ROI means the goal is to generate 14 x 1.25 = 17.5 cents profit per exposed household. In turn, this means we need to generate 35 cents extra sales revenue per household, on average. This figure comes from the assumption of a 50% gross margin, since 50% margin on 35 cents of sales is 17.5 cents, which is in turn 25% larger than the 14 cents we spend hitting each household with ads.
The advertiser then selects a control group which will not see the advertising, and a treatment group that will. And remember, it’s not as if the control group is quiet / stable in the campaign – an awful lot of unexposed households will buy the product.
So this selection of control and treatment groups, and hitting the treatment households with ads is what the researchers did, in 25 different experiments, with variations on these basic figures. To verify if a campaign reached its 25% ROI, though, they had to detect an average difference of 35 cents or more per household in sales, between a treatment and control group, when average sales per household are $7 and the standard deviation in sales across households is $75. This is just too small a difference to detect, given the massive variation in household spending.
Conclusion
Rao and Lewis concluded even with treatment groups of 200,000 consumers, these real-world experiments were quite underpowered to be able to reasonably verify if the campaigns reached the ROI target, or even if they had any effect at all. You might think, maybe it was because the target ROI was quite small – what if it was 50%, not 25%? The answer is it would not make much difference. It would mean finding a sales difference of 40 cents per household rather than 35, again with an average sales level of $7 and a standard deviation of $75. Again, the advertising effect on sales would be trivially small compared to the natural variation that occurs.
Management take-out
So the take-out is, it’s actually very difficult to measure advertising ROI – even with carefully controlled experiments – due to massive variation in baseline spending levels across households. Or as the researchers concluded themselves, “We find that even when ad delivery and consumer purchases can be measured at the individual level, linked across purchasing domains, and randomized to ensure exogenous exposure, forming reliable estimates on the returns to advertising is exceedingly difficult, even with millions of observations”.
Based on this very large study, managers should be very cautious about promises from vendors to calculate their ROI, or that guarantee a certain ROI on advertising.
The full study (lots of maths if you like that sort of thing) is at http://justinmrao.com/lewis_rao_nearimpossibility.pdf