The Golden Rule of Forecasting Kesten C. Green University of South - PowerPoint PPT Presentation

The Golden Rule of Forecasting Kesten C. Green University of South Australia Business School with J. Scott Armstrong Wharton, University of Pennsylvania Andreas Graefe LMU Munich, Germany International Symposium on Forecasting, Seoul 11:40 AM − 12:10 PM, Wednesday 26 June 2013, Room 101 Golden Rule – 2013-ISF-v12

GR is “ Be Conservative ” • Future will be like the past • Behavior • Long-run relationships • Levels • Trends • Use cumulative broad knowledge • “Shoe leather” (e.g. Broad St pump) • Involve subject matter experts • Use forecasting knowledge • Evidence-based methods • Start with no-change model How many of you agree with the GR? GoldenRuleofForecasting.com 2

Are Golden Rule violations easily spotted? Yes. A. If the description of the forecasting methods is adequate. B. If not, the Golden Rule has been violated. GoldenRuleofForecasting.com 3

Causes of Golden Rule Violations Supply of 1. Complex statistical procedures 2. Big data Demand for complex statistical procedures by 1. Clients 2. Academics GoldenRuleofForecasting.com 4

When to be conservative • All forecasting problems • Especially important if situation is • Complex • Uncertain – and • Bias is likely… • Common with • government policies • investment decisions GoldenRuleofForecasting.com 5

Consider All prior data: Example using Global temperatures GoldenRuleofForecasting.com 6

Forecasts from a complex method that ignores knowledge and data: NASA GISS *Hansen ¡ et ¡al. ¡(1988) ¡NASA ¡GISS ¡forecasts ¡vs ¡actual ¡(5-­‑year ¡rolling ¡average) ¡from ¡ ¡ http://www.kaltesonne.de/?p=4006 ¡ GoldenRuleofForecasting.com 7

Betting on conservatism: TheClimateBet.com Starting at "Tipping Point” according to Gore 0.7% 0.75% 0.5% 0.5% 0.3% 0.1% 0.25% 1981% 1986% 1991% 1996% 2001% 2006% 2011% 2016% !0.1% UAH ¡ RSS ¡ 0% 1981% 1986% 1991% 1996% 2001% 2006% 2011% 2016% !0.3% !0.25% Temp ¡ up ¡ on ¡ 28 ¡ 29 ¡ !0.5% !0.7% !0.5% previous ¡month ¡ Armstrong% Gore% Monthly%anomaly% Armstrong% Gore% Monthly%anomaly% 0.6% 0.7% Temp ¡ down ¡ on ¡ 33 ¡ 35 ¡ 0.5% 0.6% 0.4% 0.5% previous ¡month ¡ 0.3% 0.4% 0.2% 0.3% Armstrong ¡wins ¡ 67% ¡ 70% ¡ 0.1% 0.2% 0% 0.1% 2007% 2008% 2009% 2010% 2011% 2012% 2013% 2014% 2015% 2016% 2017% month ¡ !0.1% 0% 2007% 2008% 2009% 2010% 2011% 2012% 2013% 2014% 2015% 2016% 2017% !0.2% !0.1% Gore ¡error ¡larger ¡ 19% ¡ 26% ¡ !0.3% !0.2% Armstrong% Gore% Monthly%anomaly% Armstrong% Gore% Monthly%anomaly% by… ¡ 0.6% 0.7% 0.5% 0.6% 0.4% 0.5% 0.3% 0.4% 0.2% 0.3% 0.1% 0.2% 0% 0.1% 2007% 2008% 2009% 2010% 2011% 2012% 2013% 2014% 2015% 2016% 2017% !0.1% 0% 2007% 2008% 2009% 2010% 2011% 2012% 2013% 2014% 2015% 2016% 2017% !0.2% !0.1% !0.3% !0.2% Armstrong% Gore% Monthly%anomaly% Armstrong% Gore% Monthly%anomaly% GoldenRuleofForecasting.com 8

Conservative problem structuring checklist 1. Obtain and use all knowledge by: a. Obtaining all relevant information and understanding b. Decomposing the problem to best use knowledge c. Use evidence-based forecasting methods validated for the situation 2. Avoid bias in the selection of methods and data by: a. Specifying multiple hypotheses or concealing the purpose of the forecast b. Obtaining signed ethics statements before and after forecasting 3. Provide full disclosure to encourage independent audits and replications GoldenRuleofForecasting.com 9

Bias in Forecasting Example: Demand forecasts for 24 large rail transportation projects are consistently optimistic, with a median overestimate of 96 percent for traffic (Flyvbjerg) GoldenRuleofForecasting.com 10

Conservative judgmental forecasting checklist 1. Avoid unaided judgment 2. Frame questions about a given issue in various ways 3. Combine independent forecasts from 5 to 20 heterogeneous experts 4. Obtain reasons for forecasts 5. Ask experts to consider why their forecast might be wrong and to then revise. 6. Use judgmental bootstrapping 7. Use structured analogies 8. Avoid judgmental adjustments GoldenRuleofForecasting.com 11

Unaided judgments are not conservative Typical belief: “Things are different now” In study of over 27,000 political and economic forecasts made over a 20-year period, 284 experts from different fields expected the status quo to change 65% of the time. However, change from the status quo actually occurred only 51% of the time. Expert political judgment (Tetlock 2005) GoldenRuleofForecasting.com 12

Global temperature forecasting Example of a complex uncertain problem “Please forecast the missing years for the series shown on the two charts and pass to the end of the row for collection.” GoldenRuleofForecasting.com 13

Hadley global mean °C temperature anomalies showing selected half-centuries: Note similarities of A & B (B has been said to be unique and near – or past the tipping point) 25 ¡year ¡ ¡ 0.8% A ¡ forecast ¡ B ¡ period ¡ 0.6% (origin ¡−. 07°C; ¡ ¡ 0.4% slope ¡−.003 ¡ p.a.) ¡ 0.2% 0.0% !0.2% !0.4% !0.6% !0.8% !1.0% GoldenRuleofForecasting.com 14

Conservative extrapolation checklist 1. Use all valid and reliable data 2. Decompose by causal forces 3. Damp trend forecasts if the: a. situation is uncertain or unstable b. forecast horizon is longer than historical series c. trend goes outside the range of the previous data d. short and long-term trends are inconsistent e. series are contrary (trend inconsistent with causal forces) Damp seasonal factors. 4. GoldenRuleofForecasting.com 15

Simon Applies Contrary Series Rule Ehrlich “ Population Bomb ” (1968) Running out of resources => price rises, economic collapse and mass starvation by the 1990s Julian Simon Human ingenuity & free markets => resource prices decline over long term Offers bet in 1980: No change in resource prices Ehrlich & Holdren select 5 metals where prices had been rising with a 10-year horizon All prices down by 1990 GoldenRuleofForecasting.com 16

Conservative causal model checklist 1. Use prior knowledge to identify predictor variables along with their directional effects, magnitudes, and reasonable limits. 2. Damp estimated coefficients toward equal weights 3. Use diverse information, data, and models (there isn’t a perfect model). 4. Use index models when there are many important variables and much knowledge about their relationships GoldenRuleofForecasting.com 17

Conservatism via combining Combine across methods & forecasters • Incorporate more prior knowledge • Reduce effects of errors and biases – Data errors – Computational errors – Model selection errors – Biased forecasts • Error reductions, under ideal conditions, exceed 50% GoldenRuleofForecasting.com 18

Survey of experts; initial finding: Broad agreement on Guidelines • 34 responses to 24 June 2013. • 93% of 27 guidelines supported by more than 70% of experts GoldenRuleofForecasting.com 19

Survey of experts; initial finding: Controversial Guidelines For 6 of 27 Guidelines, more than ¼ of experts rejected the Guideline 1. Specify multiple hypotheses or conceal the purpose of the forecast (59%) 2. Use structured analogies (50%) 3. Damp when horizon is longer than history (29%) 4. Decompose by causal forces (28%) 5. Damp when uncertain or unstable 28%) 6. Ask experts to consider why wrong and revise (27%) GoldenRuleofForecasting.com 20

Barriers to conservative forecasting Expensive, due to need to acquire comprehensive prior knowledge (shoe leather) – Unimpressive easily understood methods – Boring forecasts lack newsworthiness Impressiveness, and obscurity of complex statistical methods and unaided expert judgment, and drama of forecasts GoldenRuleofForecasting.com 21

Conclusions Golden Rule can be unpopular as a constraint on dramatic forecasts that violate the politician’s syllogism that “Something must be done; this is something; therefore this must be done.” • Complex statistical methods and large databases harm forecast accuracy by ignoring full knowledge about situation and forecasting methods • Potential for substantial gains in forecast accuracy from adopting the Golden Rule: Forecast conservatively by holding to prior knowledge and eschewing complex un- validated methods GoldenRuleofForecasting.com 22

Possible action steps What implications for a forecasting problem with which you are familiar? How many of the 27 checklist items might be used? Or consider a public policy issue, such as “What effect would the withdrawal of government funding have on scientific research?” “Will government mandated messages and images on cigarette packs increase welfare?” GoldenRuleofForecasting.com 23

Golden Rule Checklist 1/2: (% error reduction) GoldenRuleofForecasting.com 24

Golden Rule Checklist 2/2: (% error reduction) GoldenRuleofForecasting.com 25