How easy is overfitting of investment strategy parameters and how much does overfitting inflate expectations? In their February 2016 paper entitled “Backtest Overfitting in Financial Markets”, David Bailey, Jonathan Borwein, Marcos Lopez de Prado, Amir Salehipour and Qiji Zhu introduce two online backtest overfitting tools:
- Backtest Overfitting Demonstration Tool – BODT simulates the overfitting of seasonal strategies (typical of technical analysis) to find the optimal strategy within a simulated sample of prices or actual S&P 500 Index levels by varying entry day, holding period, long or short, and stop-loss level. It runs a “movie” showing the progression of Sharpe ratio optimization. BODT then tests the optimal strategy on new (out-of-sample) data. It also provides a deflated in-sample Sharpe ratio based on the number of variations tested.
- Tenure Maker Simulation Tool – TMST simulates the overfitting of econometric strategies (typical of academic journals) by varying forecasting equation parameters to maximize predictive power within a random (unpredictable) time series. It also runs a “movie” showing progression of Sharpe ratio optimization.
By overfitting, they mean repetitive use of an historical set of market data to identify the best of many variations of a strategy. Such optimality tends to target idiosyncrasies of the historical sample rather than any general market behavior. Their goals are to show how easy it is to overfit an investment strategy and how much overfitting may inflate investment performance expectations. Based on outputs of the two simulation tools, they conclude that:
- For most BODT runs, the subsequent out-of-sample Sharpe ratio is either negative or at least much lower than the in-sample optimal Sharpe ratio. In one example, the optimal in-sample Sharpe ratio is 1.59, but the out-of-sample Sharpe ratio is -0.18.
- Sharpe ratio inflation for TMST runs tends to be more extreme than that observed for BODT runs. In other words, econometric methods are susceptible to extreme overfitting, casting doubt on investment strategies published in academic journals.
In summary, evidence from simulations indicates that overfitting of investment strategies is easy and cultivates highly inflated expectations for investment performance.
Cautions regarding conclusions include:
- As with any simulator, results are realistic only to the extent that the inputs (such as simulated price data) mimic real market behaviors.
- Some strategists take steps to mitigate overfitting bias.
See also “Insidiousness of Overfitting Investment Strategies via Iterative Backtests”, “Measuring Investment Strategy Snooping Bias” and “Snooping for Fun and No Profit”.
