How should investors assess systematic trading programs? In his August 2014 paper entitled “Evaluation of Systematic Trading Programs”, Mikhail Munenzon offers a non-technical overview of issues involved in evaluating systematic trading programs. He defines such programs as automated processes that generate signals, manage positions and execute orders for exchange-listed instruments or spot currency rates with little or no human intervention. He states that the topics he covers are not exhaustive but should be sufficient for an investor to initiate successful relationships with systematic trading managers. Based on his years of experience as a systematic trader and as a large institutional investor who has evaluated many diverse systematic trading managers on a global scale, he concludes that:
- A systematic trader’s edge comes from nimbly taking advantage of the behavioral and analytical inefficiencies of other investors in a way that competing systematic traders do not.
- The prevalent reliance on academic models for investment decisions involves some or all of the following assumptions: (1) relationships are linear; (2) variable statistics are stationary (have stable means); (3) the parts comprise the whole (no emergent properties); (4) markets are in equilibrium; (5) distributions are tame (normal), not wild (power law); and, (6) processes (behaviors) are continuous. Investors can best view markets, however, as complex and adaptive systems, invalidating these assumptions. In many cases, rule-of-thumb approaches that attempt to face facts are superior to unrealistic mathematical models.
- Trading signals:
- Should seek to exploit some relatively clear behavioral or structural inefficiency.
- Generally derive from either mean reversion or trend following.
- Usually employ the above incorrect assumptions about markets.
- Often use technical rules that inherently lag the market.
- Must balance trading frequency with trading costs.
- Risk management:
- Is more important than signal generation as a driver of value creation or destruction.
- Should focus on surviving extreme adverse events.
- Should include continual updates to incorporate new data.
- Should acknowledge volatility persistence by taking more (less) risk after successes (failures).
- Should acknowledge model limitations and volatility persistence by including a stop-loss policy.
- Should pay special attention to effects of adverse events on short positions and leverage.
- Model uncertainty:
- Is the risk of using a model that does not reflect actual market behavior.
- Generally increases with the number of assumptions and parameters used in a model.
- Generally increases with the sensitivity of outcomes to different values of parameters.
- Continuously changes in an adaptive market.
- Data snooping:
- Invites the risk of finding lucky noise rather than true (repeatable) patterns.
- Involves using the same data to test different models until finding one that works, or examining the data before developing a model and/or conducting statistical research with no theoretical grounding.
- Can be mitigated by:
- First developing a hypothesis, testing it with the simplest possible model and moving on to the next hypothesis if test results are unsatisfactory.
- Applying one of the explicit, quantitative corrections developed in recent years.
- The conventional diversification approach:
- Intends to suppress asset-specific risk, leaving the portfolio exposed only to macro (such as market, credit and interest rate) risks.
- Generally involves very large model risk due to assumptions at odds with actual market behaviors and great sensitivity to small changes in parameter values.
- Typically does not address the different ways markets interact during intervals of calm and stress.
- Properly designed backtests:
- Accelerate exploitation of good ideas.
- Should be idea confirmation tools, not idea discovery tools.
- Should use asset universes that were historically tradable, not the current market universe.
- Must include realistic execution costs.
- Should include capacity analysis.
- Once live trading begins:
- Continue running the preceding backtest model in parallel to confirm agreement.
- Resist changing the model in response to losses (less than the stop-loss level) unless there is a clear reason for changing the core idea.
- Give live performance enough time for evaluation (from several months for intra-day trading to several years for infrequent trading).
- Regarding the investment manager:
- Expect a manager who truly believes in his idea to invest much of his net worth in it.
- Sooner or later managers shift to asset gathering and relative rather than absolute performance, because it is easier to collect management fees than generate performance fees.
- A focus on management fees leads to overstatement of strategy capacity.
In summary, the key goal for all investors evaluating a systematic trading program should be to understand whether its core idea and key assumptions reflect real-world behaviors and implementation considerations. The program should be a thoughtful, consistent and scalable process, not a specific trade.
For comparison and contrast, consider “Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signals (Chapter-by-Chapter Review)”, “A Few Notes from My Life as a Quant, Reflections on Physics and Finance“, “Quantitative Finance in a Nutshell” and Avoiding Investment Strategy Flame-outs.
