Marketers are less statistical than, um, statisticians

The Corporate Executive Board had an eye-catching post on the HBR blog recently, Marketers Flunk the Big Data Test. They begin:

The big-data explosion is driving a shift away from gut-based decision making. Marketing in particular is feeling the pressure to embrace new data-driven customer intelligence capabilities. No wonder a strong appetite for data is one of the most sought-after qualities in new marketers. And yet, a recent CEB study of nearly 800 marketers at Fortune 1000 companies found the vast majority of marketers still rely too much on intuition — while the few who do use data aggressively for the most part do it badly.

We’ll save the debate about big data vs. small data for another day. This article nonetheless makes several important points:

• In today’s volatile environment, where customer behaviors are changing rapidly with new technologies, judgement from past experience is increasingly unreliable.
• Many marketing “dashboards” are tuned to low-level metrics, such as clicks and open rates, that aren’t tied to “more important measures such as customer loyalty or lifetime value” — which can result in misguided incentives.
• The data explosion has created a lot of noise — which can result in “endless fire drills” — and that a relatively small number of marketers are good at filtering it.
• Top-performing marketers today, as rated by their managers, have these qualities: “comfort with ambiguity, ability to ask strategic questions based on data, and narrow focus on higher-order goals.”

For that last point, I find myself recalling Avinash Kaushik’s keynote from the other week, that the problem is very rarely data, and that marketers would be well-advised to keep remarkable customer experiences at the top of their priorities.

However, I do have to take one exception to the CEB’s post. They lead with an indictment against marketers not being statistically savvy. They asked five statistics questions, and 44% of those interviewed got four or more answers wrong; only 6% got all five right. They followed that by stating, “it didn’t surprise us that just 5% of marketers own a statistics text book.” That sounds really bad, doesn’t it?

But ironically, this strikes me as a little fast-and-loose statistically itself.

Without giving us the questions, we have to rely on the CEB’s private assessment that they were “basic to intermediate.” But more importantly, there’s no baseline to compare against. How many of those questions would software engineers have gotten wrong? How many would professional scientists have missed? Nobel prize-winning economist Daniel Kahneman had a famous study of The Law of Small Numbers, showing that even professional researchers who rely on statistics for their careers are prone to bad statistical judgement, even with relatively simple questions.

Similarly, how many people in Fortune 1000 companies own a statistics text book? If the answer overall is, say, 2%, then you could make the statement that marketers are 150% more likely to own a statistics text book — funny how putting something in bold makes it seem more true, eh? — which frames things in an entirely different light.

Of course, I believe that marketers should increase their fluency with statistics and statistical thinking. Daniel Kahneman’s new book, Thinking, Fast and Slow, is a great place to start. Darrell Huff’s classic How to Lie with Statistics is another must-read.

But if you’re going to ask marketers to think more carefully about data, it might help to lead by example.