How can investment strategy researchers best address the randomness inherent in market data and the ability of investors/markets to adapt to changing conditions? In his September 2019 paper entitled “Tactical Investment Algorithms”, Marcos Lopez de Prado reviews three methods for testing the performance of an investment rule:
- Walk-forward (WF) tests a rule against an actual historical data series, implicitly assuming that market behaviors are neither largely random nor changing (that the rule being tested is “all-weather”).
- Resampling (RS) addresses randomness in market behaviors by assuming that resampling of past observations can usefully generate possible future price paths.
- Monte Carlo (MC) addresses both randomness and adaptation in market behaviors by simulating possible future price paths based on models of price generation derived from theory and statistical analysis of actual data.
Based on his knowledge of financial markets and testing methods, he concludes that:
- Given that markets are adaptive and investors learn from mistakes, the likelihood that all-weather (WF) rules exist is slim.
- While more difficult to overfit an RS than a WF backtest, resampling a finite history still may not generate representative price paths.
- Optimizing rules for stochastic, adaptive markets argues for testing against synthetic (modeled) datasets that include many alternative data series. Modeling price generation is no more difficult than forecasting markets.
- Four advantages of MC over WF and RS are that MC allows researchers to:
- Conduct randomized but controlled experiments.
- Test an array of price generation models to accommodate changes in market behaviors.
- Incorporate wisdom in prior beliefs, such as tenets of economic theory, so that alternative futures are more likely than those of RS. Explicit incorporation allows others to object and debate.
- Generate very long samples to achieve a target confidence level.
- Tackle rule selection as a price generation process modeling problem. Identifying the current generation process is much easier than finding a rule that works well across all possible generating processes.
In summary, Monte Carlo simulation of price paths for investment rule testing has multiple advantages over walk-forward and resampling backtest methods.
Cautions regarding conclusions include:
- Modeling price generation processes is beyond the reach of most investors, who bear fees for delegating this work to an expert (fund manager). It is not obvious that such modeling is reliably effective.
- The optimal price generation processes may be more costly to exploit than suboptimal processes.
