How serious is the snooping bias (p-hacking) derived from brute force mining of stock trading strategy variations? In their August 2017 paper entitled “p-Hacking: Evidence from Two Million Trading Strategies”, Tarun Chordia, Amit Goyal and Alessio Saretto test a large number of hypothetical trading strategies to estimate an upper bound on the seriousness of p-hacking and to estimate the likelihood that a researcher can discover a truly abnormal trading strategy. Specifically, they:
- Collect historical data for 156 firm accounting and stock price/return variables as available for U.S. common stocks in the top 80% of NYSE market capitalizations with price over $3.
- Exhaustively construct about 2.1 million trading signals from these variables based on their levels, changes and certain combination ratios.
- Calculate three measures of trading signal effectiveness:
- Gross 6-factor alphas (controlling for market, size, book-to-market, profitability, investment and momentum) of value-weighted, annually reformed hedge portfolios that are long the value-weighted tenth, or decile, of stocks with the highest signal values and short the decile with the lowest.
- Linear regressions that test ability of the entire distribution of trading signals to explain future gross returns based on linear relationships.
- Gross Sharpe ratios of the hedge portfolios used for alpha calculations.
- Apply three multiple hypothesis testing methods that account for cross-correlations in signals and returns (family-wise error rate, false discovery rate and false discovery proportion.
They deem a signal effective if it survives both statistical hurdles (alpha t-statistic 3.79 and regression t-statistic 3.12) and has a monthly Sharpe ratio higher than that of the market (0.12). Using monthly values of the 156 specified input variables during 1972 through 2015, they find that:
- Tested in isolation via a single effectiveness measure, a large number of trading signals are significantly profitable.
- Based on multiple hypothesis tests using 6-factor alphas (linear regressions), 91% (65%) of the the signals successful in isolation are false discoveries.
- Tested in isolation, only 33,881 (1.62%) of trading signals pass both alpha and regression tests. Further requiring a gross Sharpe ratio greater than that of the market leaves only 801 signals.
- Under multiple hypothesis testing, only 806 (0.04%) trading signals pass both alpha and regression tests. Further requiring a gross Sharpe ratio greater than that of the market leaves only 17 signals. These 17 signals have neither clear theoretical underpinnings nor connections to the set of published anomalies.
In summary, evidence indicates that, in the absence of effective theoretical guidelines, the likelihood of a researcher finding a trading signal that generates truly abnormal returns is extremely low.
Cautions regarding findings include:
- As noted, the study essentially assumes that financial markets theory is perfectly ineffective in pre-screening stock trading signals. This assumption may be wrong.
- Analyses employ gross stock returns. Accounting for annual portfolio reformation costs and continuous shorting costs may further reduce the number of successful strategies. Shorting constraints (lack of shares to borrow) may further exclude some successful strategies.
- Annual portfolio reformation, used in all tests above, may be far from optimal in searching for successful trading signals.
- Investors may be satisfied with lower levels of statistical significance than academic researchers.
See also:
- “Most Stock Anomalies Fake News?”
- “Remedies for Publication Bias, Poor Research Design and p-Hacking?”
- “In-sample vs. Out-of-sample Performance of 888 Trading Strategies”
- “Live Performance of Alternative Beta Products”
- “Why Smart Beta Funds Will Disappoint?”
- “Backtest Overfitting: the Movies”
- “Fixing Empirical Finance”
- “Taming the Factor Zoo?”
- “Snooping for Fun and No Profit”
- “Snooping Bias Accounting Tools”
- “Navigating the Data Snooping Icebergs”
- “Insidiousness of Overfitting Investment Strategies via Iterative Backtests”
