Superforecasting: The Art and Science of Prediction
In 2010, Advance Research Projects Activity (IARPA) started a four-year forecasting competition among thousands of ordinary people and experts to determine the core traits and behaviors that lead to the most accurate forecasting results. Here is a summary of what they found along with how we use the principles in our Clear Data Sports handicapping process.
#1 Break The Question Down Into Smaller Questions
Before making a prediction first you must present a question:
How many piano tuners are there in Chicago?
The question itself may be overwhelming and virtually impossible to make an accurate prediction in this state. But if we break the question down into smaller questions that are more manageable and easier to answer, we can then make a more accurate estimate when we bring the answers to all these questions together.
Here are some examples:
- What is the number of pianos in Chicago?
- How often is a piano tuned in a year?
- How long does it take to tune a piano?
- How many hours a year does the average piano tuner work?
- Let’s look at the first question we just presented:
- What is the number of pianos in Chicago?
These questions are still somewhat tough to answer, so let’s make it even easier to answer by breaking it down into even smaller questions like:
- How many people are there in Chicago?
- What percentage of people own a piano?
- How many institutions own a piano in Chicago?
With a little bit of research and some estimating we can create answers to these questions. Here are the results of our imaginary example.
- Estimated number of people in Chicago: 2.5 million
- Estimated percentage of people that own a piano: 1%.
- Estimated number of institutions with pianos in Chicago: 25,000.
With some simple math (2.5 million people times 1% plus 25,000) we have an estimate of 50,000 pianos in Chicago.
Continue to use this process for all your questions. Break down tougher questions when necessary into easier to answer smaller questions. Once you have finished answering your questions, consolidate your information using some basic math to create an initial prediction.
This is the foundational process Superforecasters use to predict something as accurately as possible.
Clear Data Sports Application: Rather than looking at Team A vs Team B, we investigate how Team A’s offense will do against Team B’s defense, and visa versa. Looking as far into the actual matchups as we can.
Break your question down into smaller and easier to answer questions. Then research as much as possible to get as accurate as possible answers to these questions. Estimate your answers when you can’t get exacting information. Merge your data from your smaller questions using some basic math until you have your initial prediction.
#2 Find The Base Rate
Question: How likely is it that the Renzetti family has a pet?
Current information of the family:
- They live in a small house at 84 Chestnut Avenue in New York.
- Frank Renzetti is fourty-four years old and is a bookkeeper.
- Mary Renzetti is thirty-five years old and works part time at a day care.
- They have one five year old child named Tommy.
- Frank’s mother also lives with the family.
This information may be helpful in finding answers down the road, but don’t focus on it at the start as it can bog you down and give you a less accurate estimate. In the beginning you want to downplay the situations’ uniqueness and instead group it with similar types of situations and ask a broad question about that group to get a base rate.
You would want to ask a question like:
“What percentage of American households own a pet?”
Answering a question like this gives you a base rate to start from before getting into the finer more unique details of the particular situation in question.
Clear Data Sports Application: How likely is it that Home Team A will win (given a certain situation)? We use the base rate of home win percentage of all of the Home teams in that situation as a starting point.
Ask a question about a broad group that your particular situation falls into to downplay its uniqueness and get a base rate to start from.
#3 Get Into The Details
Now that you have a base rate to start with, you can start to get down into researching the details. Again, break questions down into individual manageable components that will be easier to answer than the larger questions within the situation.
Don’t get lost in aimlessly researching countless aspects and factors. You want to create questions that are targeted and purposeful that will get you closer to a more accurate prediction in the end. So before you start to research some topic that seems to be related to your question, ask yourself, “Will the information I gather from this research help me increase the accuracy of my prediction?”
Clear Data Sports Application: We add detailed information to increase the accuracy of Home team win prediction in baseball. Information such as:
What is talent BAR number of Starting Pitcher
What is talent BAR number of opposing starter.
Talent Run Projection of Home Offense.
Talent Run projection of Away offense.
BAR talent for relief pitchers Home
BAR talent for relief pitchers Away
Further break down your situation into more manageable questions and make sure the questions you ask will aid in increasing the accuracy of your final prediction.
#4 Merge The Base Rate With The Details
Comine #2 with #3 above.
Clear Data Sports Application: Base rate is Home winning percentage. This number can be narrowed down by taking component analysis vs. win percentage on scatter plot analysis. Method can be executed with specific component analysis or combination analysis (ie. BSR probabilty projections for each game) to see which elements or combination has more correlation and therefore more predictive value.
Synthesize and merge your base rate data with your detailed research into an initial prediction.
#5 Look At Your Information From Multiple Perspectives
The Superforecaster takes their initial prediction and puts it on trial in multiple ways to make sure nothing is overlooked and it is as accurate as possible.
Get Critiques from Colleagues
Present what you’ve learned and your reasoning behind your calculations to your colleagues for additional perspectives, critique, and input.
Objectively Scrutinize Your Own Work
Write your whole analysis down. Review it objectively and scrutinize it.
Ask yourself questions like:
- Do I agree with this?
- Are there holes in this? Should I be looking for something else to fill this in?
- Would I be convinced by this if I were someone else?
Try to put some time in between your initial assessment and your critique of it. Ideally a few weeks if possible. This will give you a more fresh and objective perspective.
Re-Evaluate the Question From Scratch
Assume that your first assessment isn’t complete or not completely correct and try to assess it from a different perspective. This second estimate you get is then merged with your first estimate to get an even more accurate estimate than the initial one.
Approaching the question from these two opposing perspectives will give you different results due to each one putting you in a different frame of mind during your research and analyzation. Once you have gathered your data and created a forecast from these two different perspectives merge and synthesize their results to get a more accurate forecast.
Superforecasters don’t use just one or two of these tactics for each prediction they make. They use many of them and then repeat the process to further refine their prediction. Don’t stop forecasting after just one pass through.
Clear Data Sports Application: What is the probability that a Home team will win when their pitcher is rated in top 20% (lower good side) and they are facing an offense whose production number is in bottom 20%.
Use whatever tools you have available to assess your results; get feedback from colleagues, re-evaluate your work. Look at it from different perspectives. Gather more data. And do it all over again.
#6 Think Probabilistically
By default, most people think about outcome possibilities with only two or three settings.
- Setting 1: It is going to happen.
- Setting 2: It is not going to happen.
- Setting 3: It might happen.
Superforecasters don’t think this way. They think probabilistically, in percentages, realizing that a two or three setting approach is not very accurate and forecasts must be further refined.
The more granular and probabilistic you get in your thinking and defining your estimate, the more likely your estimate will be correct. The most accurate Supeforecasters forecast down to one-tenth of a percent. They are more accurate than their colleagues that forecast down only to single percentage points.
Clear Data Sports Application: The Basewinner line maker computes probability to the last decimal. Also, the probability formulation can be adjusted based on historical evaluation of its results.
Think probabilistically down to a granular level. Calculate your predictions down to one-tenth of a percent.
#7 Update Your Forecast When New Data Is Discovered
Coming to your forecast is not final. As new data comes to light, you must apply it and update your forecast whenever possible. Superforecasters update their forecasts much more frequently than regular forecasters and are more attentive to new data.
Clear Data Sports Application: Basewinner numbers are constantly updated during season to accurately reflect talent level at game time.
Update your forecasts frequently and when new data is discovered.
#8 Don’t Let New Data Skew Your Forecast
Though it is important to be attentive to new data, be mindful not to give it too much or too little weight when recalculating your forecast. The best way to avoid this is to update your forecast with new data often and adjust it in small increments.
Clear Data Sports Application: Clear Data numbers are constantly and carefully adjusted based on the time frame that the CDSA team has determined to have advantageous predictive value.
Update your forecasts with new data often and in small increments.
#9 Have A Growth Mindset
A growth mindset is the belief that your abilities are largely the product of effort. You can “grow” to the extent that you are willing to work hard and learn.
A fixed mindset is the belief that we are who we are and abilities can only be revealed, not created and developed.
For example someone with a growth mindset that thinks they are bad at math believes if they put the work into it, they can become better at it. But with a fixed mindset, a person believes that being bad at math is a trait of theirs that cannot be fixed so they feel it is pointless to work on it and try to improve it.
Clear Data Sports Application: The betting markets can be beat with a consistent process and the right data collection and analysis.
Have a growth mindset.
The Strongest Predictor To Becoming A Superforecaster
The data from the four-year study shows that one who is committed to belief updating when new information comes to light, and also committed to self-improvement are the strongest predictors of becoming a Superforecaster. These are followed by a distant second to intelligence.
Continue working on self-improvement and be willing to update your beliefs when new information comes to light.
These Are Not Absolutes
These traits are a broad generalization of Superforecasters. Not all Superforecasters have these traits or are strong in all of these traits. Someone weak in one area can compensate with strengths in another area. Don’t think you have to have all these traits to be good at forecasting.
Not All Questions Are Equal
Some domains and questions are easier to make a more accurate forecast than others. It is much easier to accurately forecast what the weather will be like tomorrow, than what it will be sixty days from now. Accurately forecasting what a family member will have for breakfast tomorrow is much easier than accurately forecasting who will win the Super Bowl five years from now.
Don’t treat all domains and questions as having the same difficulty to accurately forecast. Some domains are tougher to more accurately forecast due to their nature and how far out into the future the forecast is.
How To Become A Superforecaster
#1 Just Do It
You can’t just study and read to become a good forecaster. Learning to forecast requires trying to forecast.
#2 Accept Failure As An Opportunity To Learn and Grow
Practice needs to be informed practice to improve. Feedback on what you do, good and bad, is essential to improvement. Accept failure as part of the learning process instead of letting it hold you back.
#3 Analyze and Adjust
Careful and self-critical analysis of an outcome is essential to becoming better at forecasting. Be critical of yourself and your work.
#4 Continue to Persevere
Don’t stop when you make a mistake or fail in some part of your work. You must persevere. When Superforecasters make a mistake or fail, they get up, dust themselves off and get back to work.