“Be approximately right rather than exactly wrong.”
John W. Tukey
How To Improve This Important Skill (That Everyone Is Bad At)
A key skill in the realm of “Data Literacy” is the ability to estimate uncertainty. Statements like, “we expect gross profits in 2021 to be $1M” are wrong out of the gate because they are point estimates, exactingly precise and impossibly unlikely. Far more useful are estimates that capture the inherent uncertainty and state that in the form of a confidence interval, such as “we estimate with 90% confidence that profits in 2021 will be between $0.8M and $1.2M,” meaning we believe there is only a 10% chance that the real value will fall outside this range. The relative width of these confidence intervals carries important information about our uncertainty in our estimate.
The problem is that most people are really, really bad at forming those sorts of uncertainty estimates. This article will teach you how to do it with much greater accuracy.
In “The Signal and The Noise,” data scientist Nate Silver uses the tales of a professional gambler and a poker champion to illustrate a skill deemed critical to their respective successes: the ability to simultaneously hold in mind multiple hypotheses about the possible outcomes of sporting events and poker hands and update their beliefs accordingly as new data are revealed. Silver writes, “Successful gamblers—and successful forecasters of any kind—do not think of the future in terms of no-lose bets, unimpeachable theories, and infinitely precise measurements. Successful gamblers, instead, think of the future as speckles of probability, flickering upward and downward like a stock market ticker to every new jolt of information.”
Similarly, in “Superforecasting: The Art and Science of Predicting,” Philip Tetlock writes about the strategies of an unlikely but elite group of forecasters whose accuracy greatly exceeded that of supposed experts and states, “[this book is] about how to be accurate, and the superforecasters show that probabilistic thinking is essential for that.”
This skill of probabilistic thinking is crucial in virtually any context in which we need to reason about an uncertain future. Humans are notoriously prone to cognitive biases that, once we have a pet theory in mind, lead us to discount information that conflicts with that theory and more heavily weight information that supports it (see Nobel prize winner Daniel Kahneman’s “Thinking Fast and Slow” for more on this fascinating topic).
This article describes a simple process for making and updating your forecasts in a principled, consistent manner that makes maximum use of the available evidence and helps avoid harmful cognitive biases.
“Data are becoming the new raw material of business.”
— Craig Mundie
If your company is like most, these days you have more data than you know what to do with and are collecting it faster than you can imagine. IBM states the every day we create 2.5 quintillion bytes worth and several studies claim that 99+% of the world’s data was created in just the last two years.
And YOU are being asked to do more with it, to make “data-driven decisions”, whether you’re an individual contributor or the CEO, by more formally incorporating data into your decision-making processes. As McAfee and Brynjolfsson report, “companies in the top third of their industry in the use of data-driven decision making were, on average, 5% more productive and 6% more profitable than their competitors.” Businesses have recognized that “data is the new oil,” (a pronouncement credited to Sheffield mathematician Clive Humby who helped establish Tesco’s Clubcard in 1994) and those that aren’t wringing the most value from their data are going to be left in the dust.
I was training a client recently and overheard a conversation in which one analyst was helping a new team member learn to use a recently deployed strategy simulator. The new analyst had been experimenting with the simulator and the more experienced one was attempting to verify with her that the outputs were “as expected” and that “nothing counter-intuitive happened.”
The ability to learn faster than your competitors may be the only sustainable competitive advantage.
— Arie De Geus
Many years ago I learned to fly. I love aviation and after receiving my private pilot certificate, I continued to train and eventually earned a commercial multi-engine certificate and instrument rating.
Perhaps the most valuable training aid I made use of was the Frasca 142 flight simulator. The simulator allowed my instructors to throw problems of ever-increasing difficulty at me (faulty instruments, system failures, engine fires, etc.) and review my responses with me during post-flight debriefings. I could repeat and rehearse until I had an innate understanding of and reflexive response to the most dire situations. Moreover, I was able to train with safety, speed, and cost-effectiveness that could never be approached using an actual aircraft.
Imagine if you could do this with your business… How should you respond to an aggressive new entrant in your market? When should you invest in new infrastructure? How quickly should you expand into new territories? What if you could “test fly” your strategic decisions before you bet your company on the outcome?
When people make decisions, they typically use their “mental models,” sometimes augmented by other types of models like spreadsheets, to try and predict the likely outcomes of their choices, then select the optimal choice. Think of this as “mental simulation.” This works fine for simple problems, but as complexity increases, our mental models and our cognitive processes are not evolved enough to deal with issues like too much data (leading to bounded rationality), long time delays (we almost always overweight near-time phenomenon), non-linear behavior (where cause and effect are not proportional), and feedback processes (those that lead to self-reinforcing or self-modulating behaviors).