Big Ideas

Is there a practical way to mitigating data snooping bias while exploring optimal parameter values? In his February 2014 paper entitled “Know Your System! – Turning Data Mining from Bias to Benefit through System Parameter Permutation” (the National Association of Active Investment Managers’ 2014 Wagner Award winner), Dave Walton outlines System Parameter Permutation (SPP) as an alternative to traditional in-sample/out-of-sample cross-validation and other more complex approaches to compensating for data snooping bias. SPP generates distributions of performance metrics for rules-based investment strategies by systematically collecting outcomes across plausible ranges of rule parameters (see the figure below). These distributions capture typical, optimal and worst-case outcomes. He explains how to apply SPP to estimate both long-run and short-run strategy performance and to test statistical significance. He provides an example that compares conventional in-sample/out-of-sample testing and SPP as applied to an asset rotation strategy based on relative momentum. Using logical arguments and examples, he concludes that: Keep Reading

How should investors interpret findings of statistical significance in academic studies of financial markets? In the March 2014 draft of their paper entitled “Significance Testing in Empirical Finance: A Critical Review and Assessment”, Jae Kim and Philip Ji review significance testing in recent research on financial markets. They focus on interplay of two types of significance: (1) the probability of a Type I error (the probability of rejecting a true null hypothesis), with significance threshold usually set at 1%, 5% (most used) or 10%; and, (2) the probability of a Type II error (the probability of accepting a false null hypothesis). They consider the losses associated with the significance threshold, and they assess the Bayesian method of inference as an alternative to the more widely used frequentist method associated with conventional significance testing. Based on review of past criticisms of conventional significance testing and 161 studies applying linear regression recently published in four top-tier finance journals, they conclude that: Keep Reading

A subscriber flagged an apparently very attractive exchange-traded fund (ETF) momentum-volatility-correlation strategy that, as presented, generates a optimal compound annual growth rate of 45.7% with modest maximum drawdown. The strategy chooses from among the following seven ETFs:

ProShares Ultra S&P500 (SSO)
SPDR EURO STOXX 50 (FEZ)
iShares MSCI Emerging Markets (EEM)
iShares Latin America 40 (ILF)
iShares MSCI Pacific ex-Japan (EPP)
Vanguard Extended Duration Treasuries Index ETF (EDV)
iShares 1-3 Year Treasury Bond (SHY)

The steps in the strategy are, at the end of each month:

1. For the first six of the above ETFs, compute log returns over the last three months and standard deviation (volatility) of log returns over the past six months.
2. Standardize these the monthly sets of past log returns and volatilities based on their respective means and standard deviations.
3. Rank the six ETFs according to the sum of 0.75 times standardized past log return plus 0.25 times past standardized volatility.
4. Buy the top-ranked ETF for the next month.
5. However, if at the end of any month, the correlation of SSO and EDV monthly log returns over the past four months is greater than 0.75, instead buy SHY for the next month.

The developer describes this strategy as an adaptation of someone else’s strategy and notes that he has tested a number of systems. How material might the implied secondary and primary data snooping bias be in the performance of this system? To investigate, we examine the fragility of performance statistics to variation of each strategy parameter separately. As presented, the author substitutes other ETFs for the two with the shortest histories to extend the test period backward in time. We use only price histories as available for the specified ETFs, limited by EDV with inception January 2008. Using monthly adjusted closing prices for the above seven ETFs and for SPDR S&P 500 (SPY) during January 2008 through February 2014 (74 months), we find that: Keep Reading

Here is this Friday’s installment of Avoiding Investment Strategy Flame-outs, a short book we are previewing for subscribers. With this post, the book preview is complete.

Chapter 9: “Getting Expert Advice (Delegating Strategy Development)”

“Section 8-2 examines in detail the attractiveness of a short-term trading strategy offered in the quasi-advisory (“educational”) marketplace. Assessing this strategy entails considerable work, only to find that it is not attractive. This chapter covers more broadly the delegation of investment strategy development, ranging from following an expert’s public advice on market timing to deposit of funds for professional management. Such practices relieve investors (at a cost) of some or all of the burdens of learning, data collection/analysis, strategy design and disciplined implementation.

“However, such delegation entails agency issues (conflicts of interest). Potentially more than they want to help their readers/subscribers/clients earn exceptional investment returns:

• Media that present investing advice want subscription fees or attention to advertisements. Media company interest in the usefulness of what they present is arguably secondary to attracting attention. In general, contributors to free media also have motives that bias what they present (attracting their own subscribers or clients).
• Academics studying financial markets want employment (and tenure) and funding of future research. They therefore must attract the attention of peers and publishers. They often have no stake in whether their research findings are useful to investors. They do have an incentive to attract the attention of investors when making a transition to investment management.
• Expert equity analysts want employment by brokers and asset managers, and access to industry sources. The interests of their bosses may not always coincide with the interests of the clients of their bosses or other investors.
• Newsletter sellers want subscription fees. Getting the attention of potential subscribers is essential to their business model. They sometimes seek attention by uncritically presenting snooped, gross trading system results as an “educational” service.
• Financial advisors want advisory fees. They must attract the attention of potential clients. As with newsletter sellers, the font used for marketing copy is much larger than that used for the legal disclaimer.
• Investment managers, mutual fund managers and hedge fund managers want management fees, normally as a percentage of account balance. They have to get the account before they can debit the balance. They have to get the attention of a potential clients before they get the account.

“A common motive across the range of investment service providers is attention-seeking, which tends to drive offerors toward extreme representations (possible but low-probability scenarios, the tails of the distribution of potential outcomes). The most extreme representations offer the “holy grail” of amazingly large and reliable returns (appealing to investor greed) or the “safety of Noah’s ark” from impending doom (appealing to investor fear).

“Conflict-of-interest materiality persists because investors have great difficulty distinguishing luck from skill when outcomes involve a high degree of randomness.”

Here is this Friday’s installment of Avoiding Investment Strategy Flame-outs, a short book we are previewing for subscribers. Chapter previews will continue for one more Friday.

Chapter 8: “Two Analysis Regimes”

“This chapter steps through two analysis regimes via examples to illustrate avoidance and mitigation of the issues covered in Chapters 1 through 6. The first example involves a widely used technical indicator, the 10-month simple moving average, but with an investigation of whether there is more information in the average than conventionally extracted. The second example constructs in detail a portfolio-level view of a short-term trading strategy offered in the quasi-advisory (“educational”) marketplace. The purpose of the examples is to illustrate different ways that most investors can use to analyze investment strategies.

“The analysis tool is Microsoft Excel. Some or all of the steps in the examples may be useful in analyzing other potentially useful asset return indicators.”

Here is this Friday’s installment of Avoiding Investment Strategy Flame-outs, a short book we are previewing for subscribers. Chapter previews will continue for the next two Fridays.

Chapter 7: “Thinking about Taxes”

“To the extent that governments tax different kinds of income/investment returns (interest, dividends, long-term capital gains and short-term capital gains) differently, taxes may be decisive for some individuals in designing a strategy or selecting one strategy over another.

“Taxes are more personal than other investment frictions. Relevant questions include:

1. What is the investor’s expected marginal tax rate?
2. Does the investor have any capital losses carried forward from prior years that may offset future gains?
3. Are the funds in a tax-advantaged account, such as (in the U.S.): a conventional Individual Retirement Account (IRA), subject to tax rates at the time of withdrawal (whatever they may be); or, a Roth IRA, subject to no taxes at withdrawal (as the rules stand now)?

“An obvious risk to long-term strategies including assumptions about taxes is that governments may change the rules at any time.

“The next two sections explore how incorporating tax avoidance into an investment strategy might impact returns.”

Here is this Friday’s installment of Avoiding Investment Strategy Flame-outs, a short book we are previewing for subscribers. Chapter previews will continue for the next three Fridays.

Chapter 6: “Modeling at the Portfolio Level”

“Evaluating strategies based only on trade-level performance, as often presented by trading advisory (“education”) services, may mislead. Some strategies concentrate opportunities, at times identifying more trades than can reasonably be addressed with a limited amount of capital and at other times identifying no trades.

“Moreover, evaluating strategies based only on a list of closed trades, with the performance of contemporaneous open trades unknown, may mislead because open trades may be losers that at times absorb all the capital and preclude further trading.

“Modeling profitability at the portfolio level in such cases may be complicated and tedious, but is essential for understanding effects of a trading strategy on wealth. Portfolio-level modeling means carefully accounting for the allocations of all capital in a portfolio at all decision points in time series.”

Here is this Friday’s installment of Avoiding Investment Strategy Flame-outs, a short book we are previewing for subscribers. Chapter previews will continue for the next four Fridays.

Chapter 5: “Checking for Market Adaptation”

“The market is a complex system with many interacting parts, and external influences. As in other social settings, there are two aspects to market evolution: (1) adaptation to changes in external influences; and, (2) adaptation to adjust internal imbalances.

“External influences include economic forces, political shifts, monetary policies, regulatory initiatives and information technology enhancements. For example:

• Economic globalization broadens the universe of assets available in the market, but tends to increase co-movement of assets.
• Political shifts may favor one industry over another or affect portfolio-level after-tax profitability of investing.
• Loose monetary policy may favor the financial industry.
• Regulatory actions on broker fees, quote granularity, short selling and margin levels impact investment frictions (profitability of trading) and cash requirements (portfolio-level returns).
• Mass availability of historical data and investing knowledge, computing power, analysis software and real-time trading accelerate market identification of and response to all market opportunities.

“Investor adaptation to such influences is generally strategic.

“Some investors continuously strive to identify and exploit internal market imbalances (pricing anomalies) through fundamental and technical analysis, both asset-specific and marketwide. They express perceived imbalances in different ways, such as:

• Undervalued versus overvalued
• Overbought versus oversold
• Too fearful versus too complacent
• Risk-on versus risk-off
• Informed versus noise

“When many investors compete in exploiting an imbalance, they supply negative feedback that suppresses it. When more investors compete, suppression is faster. More generally and abstractly, acts of exploiting characteristics of an inferred distribution of investing returns change the distribution. (There is an extensive body of countering research that attributes perceived internal market “imbalances” to rational equilibriums based on actual, but sometimes subtle, risks. The counter-counter is a proposition that people are not even grossly rational, let alone subtly rational.)

“The following sections discuss ways to detect and deal with market adaptation.”

Here is this Friday’s installment of Avoiding Investment Strategy Flame-outs, a short book we are previewing for subscribers. Chapter previews will continue for the next five Fridays.

Chapter 4: “Accounting for Implementation Frictions”

“Investment frictions (costs) include such burdens as broker transaction fee, bid-ask spread, impact of trading (for large trades), borrowing cost for shorting, cost of leverage, costs of data, software and hardware for research, fund loads, cost of advisory services and cost of an investment manager.

“These costs vary considerably by category of investor (retail or institutional), over time, across countries and across types of assets. For example:

“Transaction fees vary by broker.

“Transaction fees are generally higher percentage-wise for small trades than large trades, and therefore for investors with small accounts than those with large accounts. Sophisticated traders may be able to suppress frictions via broker arrangements and order placement algorithms.

“Market liquidity tends to be lower in emerging markets than developed markets, generally indicating higher bid-ask spreads and impacts of trading in emerging markets.

“Both transaction fees and bid-ask spreads were generally much higher in past decades than now due to regulatory changes (ending of fixed commissions and decimalization) and technological advances (lower cost of execution and lower barrier to entry for discount brokers). This variation is problematic for long backtests.

“Frictions are generally higher percentage-wise for option trades than equity trades of similar sizes. Frictions for futures trades are comparatively low.

“Frictions for aesthetic assets such as art and wine are very large compared to those for financial assets.

“Cost of an investment manager subsumes the other costs (perhaps with economies of scale) but adds incremental fees for administration and management.

“Realistic modeling of frictions is often very difficult, especially for samples spanning long time periods. Many researchers set a goal of analyzing gross risk premiums or anomalies and therefore ignore frictions in measuring returns and alphas (returns adjusted for widely accepted risk factors). However, research findings based on net results may differ substantially from those based on gross results, to the extent of rendering realistic implementations unprofitable. The following sections cover some considerations and approaches for modeling trading frictions.”

Here is this Friday’s installment of Avoiding Investment Strategy Flame-outs, a short book we are previewing for subscribers. Chapter previews will continue for the next six Fridays.

Chapter 3: “Avoiding or Mitigating Snooping Bias”

“Snooping bias, also called mining bias and more loosely benefit of hindsight, is a notorious artificial booster of backtest performance. It takes multiple forms:

• Picking the best of many rules/indicators (strategies, models) for a given data sample
• Optimizing rule parameters for a given data sample
• Restricting a data sample to find favorable performance of a given rule
• Running an investment contest among many individuals

“A sentiment shared among researchers in stochastic fields is: “If you torture the data long enough, it will confess to anything.” Because returns are noisy (substantially random), trying many combinations of rules, parameter settings and data samples will generate strategies that outperform benchmarks by extreme good luck. A prosecutor (an investor) satisfied with false confessions is likely to lose in court (the market).

“To illustrate, Figure 3-0 depicts the net cumulative values of $1.00 initial investments in each of 12 variations of the simple asset class momentum strategy introduced in Figure 1-1. This strategy shifts each month to the one of nine asset class proxies with the highest total return over a past return measurement (ranking) interval. Most of the proxies are exchange-traded funds (ETF). The 12 variations differ by the length of the ranking interval, from one to 12 months. All variations impose a switching friction of 0.25% whenever the strategy switches funds.

“Does the top-performing variation (dotted line) represent a premium earned by extracting truly valuable information from market prices, or just the payout from being the lucky winner of a lottery? The following sections address this question.”

Figure 3-0: Performance of 12 Asset Class Momentum Strategy Variations