A very good article by Ian Lurie: Know why: How to beat marketing survivorship bias, Portent blog, 18-November-2015.
Why does marketing advice suck?
We all look for marketing tactics we can repeat. You know what I mean: Some guy writes an article about marketing strategy, and boom, everyone’s off to the races.
And 95% of the time, the tactics out of context fall flat on their metaphorical faces.
There are lots of reasons marketing blueprints can get you into trouble. The one I most worry about is survivorship bias.
...our tendency to focus on and compare to techniques, people, etc. that ‘survive.’ We miss the failures because they’re less visible. It’s a common driver of the ‘correlation does not equal causation’ cliche.
Multivariate and A/B testing helps us figure out which tactics really work. But don’t stake your campaign on those results. Use them as a guide.
Test a lot.
Collect lots of ‘passive’ data, too: Traffic, behavior, sales, whatever you can. Use it when you ask “Why did this work?”
Question why something worked, and dig well past the data.
Data can lead to survivorship bias, too, because we’re interpreting it. Never, ever forget that.
In the article Ian recommends reading Fooled by Randomness by Nassim Nicholas Taleb. I concur, highly recommended.