Are more data, higher levels of signal statistical significance and more sophisticated prediction models better for financial forecasting? In their August 2017 paper entitled “Practical Significance of Statistical Significance”, Ben Jacobsen, Alexander Molchanov and Cherry Zhang perform sensitivity testing of forecasting practices along three dimensions: (1) length of lookback interval (1 to 300 years); (2) required level of statistical significance for signals (1%, 5%, 10%…); and, (3) different signal detection methods that rely on difference from an historical average. They focus on predicting whether returns for specific calendar months will be higher or lower than the market, either excluding or including January. Using monthly UK stock market returns since 1693 and U.S. stock market returns since 1792, both through 2013, they find that:
- The trade-off between sample comprehensiveness and sample freshness is mixed.
- For the full UK sample, relatively short (~30 years) and very long (> 250 years) lookback intervals yield the best forecasting accuracies. However, the number of significant observations for very long lookback intervals is small, and associated forecasting accuracies are somewhat erratic.
- For the full U.S. sample, lookback intervals in the range 10 to 50 years work best.
- For a subsample starting 1900, a 90-year (96-year) lookback interval yields the best forecasting accuracy for the UK (U.S.).
- For the full UK sample, the highest level of signal statistical significance (1%) generally produces the most accurate forecasts, but there are some ranges of long lookback intervals for which 5% works better.
- Overall, complex signal detection methods (such as GARCH and Probit) produce no more accurate forecasts than simple regressions. The best performing method varies with lookback interval.
In summary, financial market forecasters should not assume that long samples are always better than short samples, or that complex forecasting models work better than simple models.
Conversely, results indicate that researchers can impound snooping bias into findings by trying different lookback intervals, confidence levels and forecasting models.
Cautions regarding findings include:
- The measure of forecast accuracy is directional only any may not equate to profitability of a trading strategy.
- Results may differ for measurement frequencies other than monthly.
- Findings may not apply to other aspects of UK and U.S. stock market returns.
- Findings may not apply to time series for other equity markets or other asset classes.
