Last week, I gently ribbed the Marketing Leadership Council of the CEB over their HBR post, Marketers Flunk the Big Data Test. I felt the statistics they were sharing about marketers not being very statistically savvy were themselves a little questionable.
In response, one of the authors of that post, Anna Bird, graciously reached out to me to address some of my concerns.
The HBR post stated that when they asked Fortune 1000 marketers five “basic to intermediate” statistics questions, 44% of those marketers got four or more answers wrong. Only 6% of the marketers they interviewed got all five right. The implication was that marketers lacked statistical aptitude — a serious concern in an increasingly analytical and data-driven discipline.
My immediate reaction: what were the questions?
After all, statistics can often be technical and counterintuitive. Reams of academic research have shown that people in general are very bad at reasoning statistically. Were the questions asked a fair instrument to judge marketers’ capabilities in this brave new world?
You can judge for yourself. Anna shared their questions with me and gave me permission to post them here. How many of these can you get right? (Answers at the bottom of this post — no peeking.)
- True or False: Variance measure the dispersion of data around a sample average.
- True or False: How well the sample reflects the target population is more important than how large the same is.
- True or False: Regressions with more variables are always more credible than regressions using fewer variables.
- There are two hospitals in town. One is very large; the other is very small. On a particular day, 80% of babies born in one hospital are male. For which hospital is this more likely to be true?
- A bat and a ball cost $1.10. The bat costs one dollar more than the ball. How much does the ball cost?
Okay, scroll down and peek now — how’d you do?
If you didn’t ace it, take comfort: there’s a small percentage of people on the planet who could. In fact, that was one of my other objections to the original presentation of the CEB’s findings. Sure, marketers might be bad at statistics. But if almost everyone is bad at statistics, and marketers are no different than everyone else in that regard, is this worth ringing alarms bells over?
“Our goal wasn’t to rate marketers’ data savvy relative to other individuals (since the vast majority of people get that [last] question wrong),” wrote Anna. “But just to find out whether marketing analytics actually poses risks (as well as benefits) due to everyone’s natural propensity for error when interpreting numbers. We think that this risk is under-appreciated.”
So is it possible that marketers, bad at stats as they are, are actually better than most?
“No one is great at stats/analytics, but marketers are actually better than most functions, except finance and strategy. However, marketing’s information needs are also far more complex and ambiguous than most functions — so they really need better data skills. Generally, we believe that most employees — but marketers in particular — are unprepared to take advantage of emerging technology and data. And that a bit more stats training (even just basic training) would help substantially.
All good points, and I appreciate Anna’s willingness to openly engage in this discussion. I largely agree with their core premise. But I still wonder: are these type of statistics questions measuring the right thing? Are there better kinds of questions to evaluate analytical preparedness? What do you think?
Answers:
1. True. 2. True. 3. False. 4. The smaller. 5. 5 cents.
Scott, I hit 5 out of 5 right, that’s like 50% I think! Do I have to quit my marketing job?
Seriously, most marketers SHOULD be able to get at least three of these if we think marketing has any chance to be data driven. The first question leans heavily on the language of statistics, knowing the definitions is not as important, IMO, as the principles (and I wasn’t certain I had my definitions right).
The last one is a common trick question, people fall prey to answering quickly but most people know how to figure out the right answer if they just take an extra moment.
Thanks for sharing, good to see the list. To answer your last question: I believe it is an important skill, but I don’t believe aptitude is really being tested by these questions.
I agree with Eric, but these questions at the beginning of an interview could very well lead to a conversation that will reveal value beyond the questions themselves.
Besides, they’re great fodder for tormenting colleagues, if nothing else! I just spent several minutes encouraging people to spend time on khanacademy because of this article. 😉
Agree with comments above – while these questions are more on principles of statistics than actual analysis of data, when talking stats-type questions with marketers to measure aptitude, I prefer a live discussion around questions to evaluate HOW someone thinks about data and analysis. Since the job isn’t always black and white, unilaterally declaring incompetence over 5 questions is a bit dangerous.
-Eric
As you clearly point out, taking more advantage of emerging technology and data are clear needs and the CEB article found a way to get some attention on the topic, asserting otherwise isn’t recommendable.
The more I think about it, I’m coming around:
1. Marketers *should* be able to answer these questions.
2. I was a little skeptical of the value of measuring familiarity with statistical terms, such as number of variables in a regression. But the truth is that marketers, even if they’re not statisticians themselves, are increasingly going to have to interact with specialists who speak that language. Familiarity with the basic nomenclature — as with technology speak — is going to be helpful.
3. Question #5 is a bit of a trick question, but it does remind people to check their gut with their brain — what Daniel Kahneman would call keeping System 1 in check with System 2. Given marketing’s long tradition of gut instincts, which is now colliding into the data age, this is a good lesson to learn.
4. The CEB has raised an important issue in a way that’s gotten people — at least speaking for myself — thinking and talking about it. And that’s good.
5. More analytical training for marketers at all stages of their careers seems like a wise investment.
So the question is where to find a refresher on basic stats functions and vocabulary. I saw the 2 books recommended yesterday–are those nitty gritty stats primers or more theoretical suggestions about how to apply statistics to marketing?
The books mentioned in my last post are some of my favorites, but they’re not statistics books per se. Kahneman’s book is more about cognitive biases, which explains why we’re so bad at statistical thinking and helps you to self-correct accordingly. Huff’s book is a classic, but it mostly frames how statistics are commonly abused.
Personally, I’m a big fan of the “Head First” series of books — although they’re admittedly not everyone’s cup of tea. Two to look at:
http://www.amazon.com/Head-First-Statistics-Dawn-Griffiths/dp/0596527586/
http://www.amazon.com/Head-First-Data-Analysis-Statistics/dp/0596153937/
I’ve also heard good things about “Statistics in Plain English” by Timothy Urdan, although I haven’t read it myself:
http://www.amazon.com/gp/product/041587291X/
If you want to go deeper into data analytics, Philipp Janert has a terrific (but technical and software-programmer-y) book called Data Analysis with Open Source Tools. If you think you might want to take it to the “data scientist” level, this is for you:
http://www.amazon.com/Data-Analysis-Open-Source-Tools/dp/0596802358/
Other suggestions?
That’s a terrific set of recommendations. Thank you
I agree. I think the questions are a good place to start to see how the marketing expert thinks & analysis data. Followed by some questions to see what they would actually do with the data…
My undergrad, a business degree with a specialization in public relations, had a full-credit mandatory course in statistics and probability. (I also took calculus as a first-year elective but that’s another story.) I had no trouble with the questions.
Even so, I would question the premise that marketers should be able to answer these. Terms like data dispersion, variance and regression are a bit esoteric, and I’d expect my analyst to be able to tell me what it all meant, rather than just report the raw numbers and degrees of variance. My job as a marketer would be to then develop the best strategies given the data analysis I was provided.