What methods are available to suppress data snooping bias derived from testing multiple strategies/strategy variations on the same set of historical data? Which methods are best? In their March 2018 paper entitled “Systematic Testing of Systematic Trading Strategies”, Kovlin Perumal and Emlyn Flint survey statistical methods for suppressing data snooping bias and compare effectiveness of these methods on simulated asset return data and artificial trading rules. They choose a Jump Diffusion model to simulate asset return data, because it reasonably captures volatility and jumps observed in real markets. They define artificial trading rules simply in terms of probability of successfully predicting next-interval return sign. They test the power of each method by: (1) measuring its ability not to choose inaccurate trading rules; and, (2) relating confidence levels it assigns to strategies to profitabilities of those strategies. Using the specified asset return data and trading rule simulation approaches, they conclude that:
- Principal statistical methods for suppressing data snooping bias are:
- Controlling Family-Wise Error Rate (FWER), specified as probability of making at least one false discovery.
- Controlling False Discovery Rate (FDR), specified as expected proportion of false discoveries.
- White’s Reality Check (WRC), which applies bootstrapping to generate unbiased confidence levels.
- Superior Predictive Accuracy (SPA), similar to WRC but using a different test statistic and a sample-dependent distribution.
- Monte Carlo Permutation (MCP), determining whether an informed strategy is significantly superior to a strategy with no predictive power, with the distribution of the latter generated via bootstrapping.
- Corradi and Swanson (CS), building on WRC by evaluating forecast combinations via examination of quantiles of an expected loss distribution.
- Step-M, a step-wise extension of WRC.
- General findings regarding data snooping bias are:
- Bias is highest for trading rules with 50% probability of correctly predicting asset return sign.
- The higher the volatility of simulated returns, the greater the bias. This effect is strong compared to other sources of bias.
- The more outliers (jumps) of simulated returns, the greater the bias. This effect is weak compared to other sources of bias.
- The more strategies tested, the greater the bias.
- The longer the backtest, the lower the bias.
- All methods substantially suppress data snooping bias compared to the simple and widely used t-test. WRC and MCP perform best. Step-M is appropriate when WRC and MCP are too conservative.
In summary, evidence from simulated asset returns and artificial trading rules indicate that White’s Reality Check and Monte Carlo Permutations are the most effective methods for suppressing data snooping bias when testing multiple strategies.
Cautions regarding conclusions include:
- Methods are beyond the reach of most investors, who would bear fees for delegating to an investment manager.
- Results for real market data and real trading rules with frictions may differ from those above for simulated asset returns and artificial trading rules.
- Data snooping bias suppression methods require discipline in tracking the number of strategies/strategy variations considered.
- Borrowing strategies/strategy features based on research by others obscures the number of alternatives considered (secondary snooping bias).
