How do you know what your data is actually telling you?

by Kate Pavao

Big data is a big buzzword, but it is easy for executives to get lost in the numbers—and make business decisions based on bad analysis.

In Kaiser Fung’s forthcoming book Numbersense, the statistician and blogger — who published Numbers Rule Your World: The Hidden Influence of Probabilities and Statistics on Everything You Do in 2010 — examines diverse analytical problems, from how law schools manipulated data to get better U.S. News and Reports law school rankings to when a Groupon deal makes sense for a merchant.

Here, Fung talks to Profit about what’s next for big data, how executives can use the stories in his book to develop their own Numbersense—and questions you should ask about every statistic you see.

Profit: It’s been three years since Numbers Rule Your World was published. How has the landscape changed since then?

Fung: The analytics and data science area of business has grown a lot. It’s just amazing how much attention is being paid to it now. Clearly, the overall awareness of how data can impact business is way higher than before.

Right now, business leaders are asking about how much data they can process — and how quickly they can do it. In the next stage. there will be more and more pressure to get results rather than just building out infrastructure.

Their questions will be, “Now that we’ve made all these investments, what’s actually changed? Are we able to see the impact in our business metrics? Are we getting smarter? Can you point to a specific learning that the data gives us that we would have otherwise not been able to get?”

The fact that business leaders have so much more data means that they are asking a lot more questions, which is already a really good thing. These questions are shedding light in areas of the business that in the past have been quite out of reach.

The most successful businesses are going to be able to able to prioritize the work and focus attention on projects that have the potential to deliver large amounts of value, One downside of having so much data is that business leaders could very easily get stuck in looking at interesting little patterns — but then miss the big picture.

Profit: What is Numbersense?

Fung: Any really great data analyst is going to have Numbersense now, which is this ability to sniff out data interpretations. It is a little bit of an intangible thing: It means when you look at a data interpretation and analysis, you put quotes around it and know which questions to ask. You have some sense of which part is supported by evidence and which part is potentially theory and assumption.

Profit: Why do executives need it?

Fung: Executives are one of the biggest consumers of big data. In their daily lives, they are going to see more and more of their employees come to them with arguments that are supposedly supported by the data. So assuming that most CEOs are not data analysts or math majors, it’s really important for them to also have this ability to push back. They need to actually understand what their people are showing them and make smarter decisions about what data analysis can be trusted.

Profit: How can executives start to develop some Numbersense?

Fung: A lot of Numbersense is learned through an apprenticeship model.

In my book, I take all these contemporary examples that have shown up in the news over the last five years, and then I walk readers through how I think about these issues, and how I came to the conclusion of whether I liked the analysis or not. I am hoping that my readers will pick up a thing or two from my book about how I approach data interpretations.

Another way is to follow writers and bloggers who have Numbersense and read their work. Through the process of immersion, executives can pick up how these experts approach data analysis, and improve that way the way they do their own.

Profit: Is the first step just to question every statistic you read?

Fung: That’s part of it. Once you’re in a world of so much data, people can either maliciously or just inadvertently take certain parts of the data, and tell a story that way. So you could end up with analysis that is not the full picture.

If you have Numbersense you’ll hopefully be able to see through some of the hype and be able to ask the right questions — because you know what the questions are. This way you can find data that’s potentially missing from the presentation to complete the picture.

I also emphasize really doing the legwork to understand how the data is collected and how metrics are being defined. This means you will have to read glossaries and other dry material, but that’s how you get smart. Oftentimes the most important pieces of information are in the footnotes, references and other hidden places.

Profit: When it comes to questions, do you need to think about who has something to gain from a particular data analysis?

Fung: That’s always a good thing to be wondering about because that typically does effect what kind of information you end up seeing.

Profit: Ultimately what do you want executives know?

Fung: It will become more and more important for CEOs and C-level staff to be able to handle quantitative analysis. When they see a piece of data analysis, they need be able to figure out what questions to ask, and be able to figure out the strengths and weaknesses of that analysis.

Data can lead them down a good path, but it can also lead them down the wrong path very easily, and this gets worse when there is more data out there. Big data can be great thing if they use that tool properly, and it can be harmful if they are unable to figure out if the analysis in front of them is actually correct or not.

Numbersense will be released on July 12.

Kate Pavao is a frequent contributor to Profit.