What are common cautions regarding exploitation of academic and practitioner papers on financial markets? To investigate, we collect, collate and summarize our cautions on findings from papers reviewed over the past year. These papers are survivors of screening for relevance to investors of a much larger number of papers, mostly from the Financial Economics Network (FEN) Subject Matter eJournals and Journal of Economic Literature (JEL) Code G1 sections of the Social Sciences Research Network (SSRN). Based on review of cautions in 109 summaries of papers relevant to investors posted during mid-March 2018 through mid-March 2019, we conclude that:
Most of the 109 papers summarized are academic. Most quantify performance of some investment factor or strategy, but some are discussion-only. On average, there are about three cautions on findings per paper. Cautions fall into the following categories:
- Ignoring or not rigorously accounting for implementation costs (70% of papers):
- Costs most often pertain to periodic portfolio rebalancing and shorting (cost of borrowing shares with loan supply sometimes limited or absent). These costs are admittedly very difficult to address over long sample periods.
- Some papers address costs but assume trading frictions that are likely unachievable by most investors.
- This issue undermines exploitation of nearly all factor studies.
- The issue is most acute when performance derives from illiquid assets (including small-capitalization stocks, especially for equal-weighted portfolios) and high portfolio turnovers. Potential concentration of turnover in times of market stress (illiquidity) exacerbates.
- The issue is also of import when comparing factors/strategies that have different liquidity dependencies and turnovers.
- Some papers present strategies involving material costs of data, software and programming expertise.
- Indications of uncorrected data snooping bias (40% of papers):
- Many papers conduct snooping of samples, models/strategies and parameter values without correcting for bias, focusing on the best (lucky) outcomes.
- Some appear to inherit bias by borrowing setups snooped in prior research.
- As a special category of costs, indications of complexity requiring delegation to an investment or fund manager (about 35% of papers):
- Some factors/strategies require complex data inputs and software development. Many require periodic execution of large numbers of trades.
- Many factors/strategies require large amounts of capital to diversify across enough assets to assure outcome reliability.
- Investors who must delegate these tasks bear attendant administrative and management fees.
- Sampling issues (about 50% of papers):
- Some papers employ sample periods that are short in terms of variety of market and economic conditions, and/or in terms of the longest measurement intervals for input and output variables. Dependence on good bond returns for a sample since the 1980s (secular interest rate decline) is a recurring issue.
- Some papers seek to extend sample periods or avoid single-path dependence by simulating asset prices.
- Simulation requires arguable assumptions about how markets price assets.
- Extending sample periods introduces the impossibility of accounting for how the market might have adapted to actual availability of hypothetical assets.
- Findings from some papers depend critically on rare/extreme events, such as October 1987 and 2008-2009 equity market crashes.
- Conversely, some papers exclude rare/extreme events (hyperinflation or market crashes) to avoid impact of outliers, but outliers happen.
- Some papers invite survivorship bias by assuming that long histories available for some markets, funds or stocks are representative of all.
- Some papers imply that results for one asset or asset class (such as for technical trading rules) apply to others.
- Some papers (e.g., on cryptocurrencies) rely on data for immature and rapidly evolving markets.
- Some papers conduct in-sample analyses of as-revised economic and sentiment data, impounding look-ahead bias from periodic recalibrations to preserve a set mean, revise seasonality adjustments or reflect modeling changes.
- Methodology issues (about 66% of papers)
- By implication of emphasis on average returns and standard deviations of returns, many papers assume tame return distributions.
- Papers based on surveys face inherent uncertainty about representativeness of respondents and about relationships between actual behaviors and survey responses.
- Some papers focus on in-sample results or inputs, with attendant look-ahead bias.
- Some papers emphasize statistical significance with little or no attention to economic significance.
- A few papers test strategies that generate just a few signals/events over long periods.
- A few papers employ models that generate overlapping signals, with an implied need for cash reserves at the portfolio level.
- A few papers employ overlapping measurements to generate inputs and/or results without correcting for attendant autocorrelations.
- Some papers fail to recognize results as tail effects (economic significance driven by extreme observations) rather than broader linear effects.
- A few papers require forecasts of input variables, compounding complexities and uncertainties.
- A few papers use lazy/easy benchmarks unrepresentative of the strategy asset universe.
- Miscellaneous issues (about 20% of papers) such as:
- A few otherwise interesting papers are vague about strategy setup.
- A few papers fail to untangle complexity to isolate sources of effectiveness (such as combining cyclically adjusted price-to-earnings ratio and momentum, without testing them separately).
- A few papers ignore latent risks (such as leveraged fund offeror solvency).
In summary, investors should be skeptical about transfer of findings from formal papers on financial markets to investment practices, especially with regard to implementation costs, direct and inherited data snooping, model/strategy complexity, and rigor in sampling and methodology.
Cautions regarding conclusions include:
- Some categories of cautions do not apply to some papers. For example:
- The percentage of papers covering pricing factors that do not rigorously account for implementation costs is closer to 100% than 70%.
- Papers addressing the issue of data snooping do not engage in uncorrected data snooping.
- Some categories of cautions apply more to academic papers (accounting for implementation costs) and others apply more to practitioner papers (methodology issues).
- Due to data/modeling constraints, we plead guilty to some modest transgressions as summarized above in our own studies.
- Other reviewers may identify other issues.
