Data snooping bias is pervasive in published research and quantitative investment strategies. Should investors resign themselves to the consequence that investment managers/funds offer products picked mostly on past luck? In his May 2018 presentation package entitled “How the Sharpe Ratio Died, and Came Back to Life”, Marcos Lopez de Prado introduces an approach to Sharpe ratio estimation via backtesting that would enable academia, regulators and investors to distinguish between strategies that probably work and those that probably do not. Based on the evolution of Sharpe ratio estimation approaches, he concludes that:
- During 1966-2012, researchers expanded robustness of Sharpe ratio as a strategy performance metric:
- From initially requiring that returns are both independent and identically distributed (IID) and normally distributed.
- To IID and non-normally distributed (accounting for skewness and kurtosis).
- To just stationary and ergodic non-normally distributed.
- To estimating a minimum sample size for a given Sharpe ratio threshold.
- However, by 2014, backtest overfitting (data snooping) is so severe that Sharpe ratio is practically dead as a strategy backtest evaluation tool, with most quantitative firms investing in false discoveries. Correcting for effects of data snooping on expected Sharpe ratios is impractical due to difficulty of accounting for inherited (via past research) and new backtest trials.
- As of 2018, a tractable method is available to estimate data snooping by:
- Assuming that actual backtesting is close to exhaustive.
- Recursively identifying empirically independent clusters of known backtest trials, based on pairwise correlations of backtest trial returns.
- Using straightforward mathematics to correct the best backtested Sharpe ratio (i.e., the one presented by offerors) for the number of effectively independent backtests. Monte Carlo simulation confirms that this approach works.
In summary, investors should demand that investment managers/funds address quantitatively and robustly the effects of data snooping on offered active investment strategies.
Cautions regarding conclusions include:
- Use of Sharpe ratio in the investment community remains largely anchored to its early, theoretically constrained definition (IID and normal).
- Though tractable, the proposed approach is not simple and easy from an investor perspective.
- Institutional incentives for academics, journals and investment management firms work against the author’s advice.
This paper is a presentation-style version of “Estimating the Level of, and Correcting for, Snooping Bias”, distilling the author’s points of emphasis.
