“Be approximately right rather than exactly wrong.”
John W. Tukey
How To Improve This Important Skill (That Everyone Is Bad At)
A key skill in the realm of “Data Literacy” is the ability to estimate uncertainty. Statements like, “we expect gross profits in 2021 to be $1M” are wrong out of the gate because they are point estimates, exactingly precise and impossibly unlikely. Far more useful are estimates that capture the inherent uncertainty and state that in the form of a confidence interval, such as “we estimate with 90% confidence that profits in 2021 will be between $0.8M and $1.2M,” meaning we believe there is only a 10% chance that the real value will fall outside this range. The relative width of these confidence intervals carries important information about our uncertainty in our estimate.
The problem is that most people are really, really bad at forming those sorts of uncertainty estimates. This article will teach you how to do it with much greater accuracy.
Benchmark Yourself: A Capen Quiz
How bad are we at doing this? Try this fun little quiz, shamelessly poached from the blog of my brilliant former colleague Dr. Tom Fiddaman. For each of the questions below, estimate a lower and upper bound that you think captures the true answer with 90% probability. For example, if trying to estimate the weight of my car, I might come up with a range of between 3400 and 4100 pounds (the actual curb weight turns out to be 3550, so it fell within my range albeit close to one end). No fair Googling and don’t give ridiculous ranges like “between 0 and 1000000 pounds.” Write down your responses and compare to the correct answers at the end. In the ideal case, if you’re estimating 90% confidence bounds, you should get nine out of ten correct.
- What is the wingspan of an Airbus A380-800 superjumbo jet?
- What is the mean distance from the earth to the moon?
- In what year did the Russians launch Sputnik?
- In what year did Alaric lead the Visigoths in the Sack of Rome?
- How many career home runs did baseball giant Babe Ruth hit?
- How many iPhones did Apple sell in FY 2007, its year of introduction?
- How many transistors were on a 1993 Intel Pentium CPU chip?
- How many sheep were in New Zealand in on 30 June 2006?
- What is the USGA-regulated minimum diameter of a golf ball?
- How tall is Victoria Falls on the Zambezi River?
Most people tend to be overconfident in their estimates, i.e. their ranges are too narrow. Parenthetically, anecdotal evidence suggests that the higher one is in the org chart, the more overconfident their estimates become.
The Casino Method
This simple mental exercise will help you think more critically about your estimates and improve your calibration of the uncertainty. Imagine you’re in a casino standing before a “Wheel Of Fortune” with ten possible outcomes labeled 1 through 10. When you bet on this game, you win if it lands anywhere from 1 to 9 and lose if it lands on 10. Pretty good odds, right?
Now imagine you’re offered the choice of either (a) betting on the Wheel Of Fortune or (b) betting on whether or not your 90% confidence interval correctly captures the true answer. Maybe you’ve estimated an interval of 95,000 to 105,000 widgets for some scenario? Place your bet!
If your instinct is to choose the wheel, you are probably under-confident in your estimate. That is, you believe your estimated interval embraces the true answer with less than 0.9 probability and thus is too narrow.
On the other hand, suppose you have a really wide estimate and your wager is now looks like this:
If your instinct is to bet on your estimated interval rather than the wheel, you are implicitly saying you believe your estimate has higher than 90% probability of capturing the true value and thus is too broad.
Here’s the trick: iteratively think about which bet you’d make and refine your estimate until you reach a point where you don’t have a preference for one bet over the other. That is, you believe the probability of winning either bet is equally likely, 90%.
Resources
I first learned of this approach from Doug Hubbard’s phenomenal book, “How To Measure Anything: Finding The Value Of Intangibles In Business.” It’s one my top-recommended books for any business decision maker.
Capen Quiz Answers
- 79.75 meters (261 feet 8 inches) [link]
- 384,400 kilometers (238,855 miles) [link]
- 1957 [link]
- 410 AD [link]
- 714 [link]
- 1,389,000 [link]
- 3,100,000 [link]
- 40.1 million [link]
- 43 mm (1.68 inches) [link]
- 105 meters (344 feet) [link]
How did you do? Leave your score in the comments!