Do motives of financial market researchers justify strong skepticism of their findings? In his brief August 2021 paper entitled “Be Skeptical of Asset Management Research”, Campbell Harvey argues that economic incentives undermine belief in findings of both academic and practitioner financial market researchers. Based on his 35 years as an academic, advisor to asset management companies and editor of a top finance journal, he concludes that:
- Given widespread experience that true risk-adjusted outperformance (alpha) is very rare, how can 90% of articles published in finance journals report evidence in support of such outperformance?
- Academic researchers have powerful incentives (promotion and tenure) to produce papers that attract citations. Positive findings garner far more citations than null results.
- Researchers often resort to exhaustive, undisciplined data snooping to obtain positive findings. For example, they may:
- Consider many variables and cherry pick the best.
- Consider different sample start dates and pick the one with the best result.
- Exclude disruptive outliers such as the 2008 global financial crisis or the 2020 COVID-19 crash.
- Transform variables (as when using logarithmic values or applying volatility scaling) to improve fit.
- Consider different fitting methodologies (such as weighted least squares rather than simple regression) to boost significance.
- Although statistical methods to correct for data mining exist, researchers rarely use them.
- It is practically impossible for editors, peer reviewers and investors to detect/quantify data snooping in submitted papers.
- The data snooping problem is somewhat less severe among practitioners because asset managers who overfit their backtests tend to: (1) underperform in live trading and thus generate no performance fees; and, (2) lose investors and incur reputational damage. Even so, there is a lot of low-quality practitioner research aimed at inattentive investors.
- Investors should therefore:
- Be very skeptical of both academic and practitioner research findings.
- Take the research culture into account (ask about data snooping).
- Try to quantify the costs of selecting bad research versus good research.
- Question each strategy’s economic foundation, with extra skepticism for supporting stories concocted post-discovery.
- Ask about what did not work, with extra suspicion for failures not previously reported (a newly seen cockroach probably means many unseen).
In summary, strong motives among financial market researchers to publish positive findings should make investors skeptical of all of it.
As noted in the paper, this conclusion applies to fields other than finance.
