Here is this Friday’s installment of Avoiding Investment Strategy Flame-outs, a short book we are previewing for subscribers. Chapter previews will continue for the next six Fridays.
Chapter 3: “Avoiding or Mitigating Snooping Bias”
“Snooping bias, also called mining bias and more loosely benefit of hindsight, is a notorious artificial booster of backtest performance. It takes multiple forms:
- Picking the best of many rules/indicators (strategies, models) for a given data sample
- Optimizing rule parameters for a given data sample
- Restricting a data sample to find favorable performance of a given rule
- Running an investment contest among many individuals
“A sentiment shared among researchers in stochastic fields is: “If you torture the data long enough, it will confess to anything.” Because returns are noisy (substantially random), trying many combinations of rules, parameter settings and data samples will generate strategies that outperform benchmarks by extreme good luck. A prosecutor (an investor) satisfied with false confessions is likely to lose in court (the market).
“To illustrate, Figure 3-0 depicts the net cumulative values of $1.00 initial investments in each of 12 variations of the simple asset class momentum strategy introduced in Figure 1-1. This strategy shifts each month to the one of nine asset class proxies with the highest total return over a past return measurement (ranking) interval. Most of the proxies are exchange-traded funds (ETF). The 12 variations differ by the length of the ranking interval, from one to 12 months. All variations impose a switching friction of 0.25% whenever the strategy switches funds.
“Does the top-performing variation (dotted line) represent a premium earned by extracting truly valuable information from market prices, or just the payout from being the lucky winner of a lottery? The following sections address this question.”
Figure 3-0: Performance of 12 Asset Class Momentum Strategy Variations
