Big Ideas

Are any trading strategy backtest performance statistics predictive of out-of-sample results? In their March 2016 paper entitled “All that Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms”, Thomas Wiecki, Andrew Campbell, Justin Lent and Jessica Stauth compare backtest and out-of-sample performance statistics for 888 algorithmic trading strategies. They first screen a larger set of strategies to remove duplicates, outliers and algorithms unlikely to represent real strategies. They next test the selected strategies in-sample (IS) with data that was available to the developers (from 2010 through deployment dates between January and June in 2015). They then test the strategies out-of-sample (OOS) during June 2015 through February 2016. All tests employ minute-by-minute prices for trade entry/exit and include robustly estimated trading frictions. Performance metrics derive from end-of-day positions/prices. Most tests are linear regressions relating individual IS-OOS performance metrics (such as Sharpe ratio). They also examine abilities of several multivariate machine learning techniques to predict performance, ultimately via an an equal-weighted portfolio of the 10 strategies predicted to have the highest OOS Sharpe ratios. Using position and price data for the 888 strategies during the specified IS and OOS periods, plus the total number of backtest days actually employed by each strategy developer, they find that: Keep Reading

Do asset price series in general reliably exhibit trends and reversals? In his May 2016 paper entitled “Trend, Mean-Reversion or Random Walk? A Statistical Analysis of Price Behavior in Major Markets”, Theo Athanasiadis tests a wide variety of financial market price series for existence of significant trends and reversals. He considers both spot and futures price series in U.S. dollars for 56 major markets: 16 developed equity market indexes; the S&P 500 implied volatility index (VIX); 25 liquid commodities covering all basic sectors; 5 liquid currency exchange rates versus the U.S. dollar; and, 9 liquid government bonds of varying durations. For futures contract returns, he uses the most liquid contracts (typically nearest or next-nearest) and rolls accordingly. He employs three statistical tests of time-series behavior: autocorrelation, variance ratio and positive/negative runs relative to median. He considers weekly, monthly, quarterly and semiannual returns in both univariate and multivariate tests. Using spot and futures price returns at the specified frequencies for all 56 markets as available during January 1999 through March 2016, he finds that: Keep Reading

Are the backtests provided for alternative beta investment products representative of their future live performance? In their March 2016 paper entitled “Quantifying Backtest Overfitting in Alternative Beta Strategies”, Antti Suhonen, Matthias Lennkh and Fabrice Perez compare the backtested and live performances of alternative beta products offered by investment banks. The strategies underlying these products are formulaic and non-discretionary, designed to extract risk factor/style premiums (such as value, momentum, carry or term) or exploit some financial market anomaly (such as turn-of-the-month or mean reversion). Specifically, they:

1. Present an overview of alternative beta products offered by investment banks.
2. Compare backtested and live performance data for these products.
3. Compare  backtested and live factor exposures four four strategy families (equity value, equity volatility, fixed income term and currency exchange carry).

Using daily returns in U.S. dollars of 215 alternative beta strategies across five asset classes and 11 strategy types offered by 15 investment banks as available during January 1990 through early March 2015, they find that: Keep Reading

What happens out-of-sample to stock portfolios with weights derived from extreme in-sample fitting? In their February 2016 paper entitled “Stock Portfolio Design and Backtest Overfitting”, David Bailey, Jonathan Borwein and Marcos Lopez de Prado examine backtest overfitting in the context of designing a stock portfolio/fund. Their test approach is:

1. Construct split-adjusted, dividend-reinvested price series for all S&P 500 components as of January 22, 2016 with continuous monthly prices during 1991 through 2015 (277 stocks).
2. Select a target performance profile, including annualized return (6%, 8%, 10%, 12% 15% or 18%) and “shape” of return (principally, steady increase every month).
3. Apply an optimization program to determine the fixed stock price weights (0.1% increments) that achieve target performance profile in-sample during 1991-2005 (requiring monthly rebalancing of the portfolio to those weights).
4. Apply these stock price weights during 2006-2015 (again, with monthly portfolio rebalancing) to measure out-of-sample performance.

In initial tests, they allow negative weights (shorting). Because of the risks of shorting, they repeat analyses with a long-only constraint. They note that their in-sample fitting process considers “an inconceivably large set” of possible weights. They use the S&P 500 Total Return Index as a benchmark. Using adjusted monthly prices for the specified stocks from the end of December 1990 through the beginning of January 2016, they find that: Keep Reading

How can investors easily estimate the degradation from optimized in-sample Sharpe ratio to out-of-sample expected Sharpe ratio? In their February 2016 paper entitled “Noise Fit, Estimation Error and a Sharpe Information Criterion”, Dirk Paulsen and Jakob Sohl derive a simple correction for the upward bias in an optimized in-sample Sharpe ratio. The upward bias derives from fitting: (1) random noise within the backtest sample; and, (2) peculiarities in the backtest sample that make it less than perfectly representative of the entire (unknowable) series. In other words, even if no predictability exists, fitting noise “discovers” some. And, even if predictability exists, predictability within a backtest sample will likely be different from predictability in the entire series. Based on derivations addressing quantification of these two sources of bias, they conclude that: Keep Reading

How easy is overfitting of investment strategy parameters and how much does overfitting inflate expectations? In their February 2016 paper entitled “Backtest Overfitting in Financial Markets”, David Bailey, Jonathan Borwein, Marcos Lopez de Prado, Amir Salehipour and Qiji Zhu introduce two online backtest overfitting tools:

1. Backtest Overfitting Demonstration Tool – BODT simulates the overfitting of seasonal strategies (typical of technical analysis) to find the optimal strategy within a simulated sample of prices or actual S&P 500 Index levels by varying entry day, holding period, long or short, and stop-loss level. It runs a “movie” showing the progression of Sharpe ratio optimization. BODT then tests the optimal strategy on new (out-of-sample) data. It also provides a deflated in-sample Sharpe ratio based on the number of variations tested.
2. Tenure Maker Simulation Tool – TMST simulates the overfitting of econometric strategies (typical of academic journals) by varying forecasting equation parameters to maximize predictive power within a random (unpredictable) time series. It also runs a “movie” showing progression of Sharpe ratio optimization.

By overfitting, they mean repetitive use of an historical set of market data to identify the best of many variations of a strategy. Such optimality tends to target idiosyncrasies of the historical sample rather than any general market behavior. Their goals are to show how easy it is to overfit an investment strategy and how much overfitting may inflate investment performance expectations. Based on outputs of the two simulation tools, they conclude that: Keep Reading

Has the Moore’s Law-driven advance in financial information technology strengthened the hand of Murphy’s Law in markets? In the January 2016 version of his paper entitled “Moore’s Law vs. Murphy’s Law in the Financial System: Who’s Winning?” Andrew Lo reviews big unintended consequences of technology-leveraged finance including fire sales, flash crashes, botched initial public offerings (IPO), catastrophic algorithmic trading errors and access failures. He then discusses the counterbalancing roles of technology in elevating and suppressing financial system risk. Based on a survey of recent financial system breakdowns and his experience, he finds that: Keep Reading

Early in the first chapter of their 2015 book, Superforecasting: The Art and Science of Prediction, Philip Tetlock and Dan Gardner state: “…forecasting is not a ‘you have it or you don’t’ talent. It is a skill that can be cultivated. This book will show you how.” Based on the body of research on forecasting (with focus on Philip Tetlock’s long-term studies), they conclude that: Keep Reading

Wesley Gray, Jack Vogel and David Foulke preface their 2015 book, DIY Financial Advisor: A Simple Solution to Build and Protect Your Wealth, by stating that: “This book is a synopsis of our research findings developed while serving as a consultant and asset manager for large family offices. …Our book is meant to be an educational journey that slowly builds confidence in one’s own ability to manage a portfolio. In our book, we explore a potential solution that can be applicable to a wide variety of investors, from the ultra-high-net-worth to middle-class individual, all of whom are focused on similar goals  of preserving and growing their capital over time.” Based on their research, they conclude that: Keep Reading

What are pros and cons of extending the definition of financial index beyond conventional market capitalization (buy-and-hold) weighting? In the October 2015 draft of his paper entitled “What Is an Index?”, Andrew Lo proposes that any portfolio satisfying three properties should be considered an index: (1) transparent (public and verifiable); (2) investable (realistic and liquid benchmark); and, (3) entirely rules-based (allowing no judgment/discretion). He calls indexes that are not weighted by market capitalization dynamic indexes (requiring frequent rebalancing). He distinguishes between active investing and active risk management. He also addresses the elevated risk of snooping bias as dynamic indexes proliferate. Based on a broader perspective on indexes, he concludes that: Keep Reading