Are there intractable weaknesses of historical inference as a tool to predict the behaviors of financial markets? In the May 2015 draft of his article entitled “Beyond Backtesting: The Historical Evidence Trap”, Ulrich Hammerich briefly describes four weaknesses of backtesting more difficult to address than overfitting/snooping, neglect of trading frictions and data quality. He calls these weaknesses the technological trap, the market efficiency trap, the publication trap and the affiliation trap. Based on common sense arguments and references to some past research, he concludes that:
- Market patterns and trading strategy performances are unstable due to rapid technological advances. With access to today’s information and computing power, past investors and markets would have behaved differently. Similarly, with access to further technological advances, future investor and market behaviors will evolve.
- Falling transaction costs, increasing transaction speeds/volumes and growing access to market data boosts market efficiency, thereby progressively weakening the performances of all historically successful trading strategies.
- Publication of a successful trading strategy attracts users, and increased competition for the strategy’s premium tends to extinguish the premium.
- Incentives for practitioners, academic researchers and peer reviewers to publish only successful strategies cultivates a sense that outperformance is easier than it is.
In summary, historical inference (backtesting) has unquantifiable weaknesses when applied to complex systems with feedback (social systems such as financial markets).
For summaries of some relevant empirical research, see “Calibrating Ancient History”, “The Vanishing Bid-Ask Spread and Market Efficiency” and “Effects of Market Adaptation”.
