Guru Stock Market Forecasting Grades

Can experts, whether self-proclaimed or endorsed by others (publications), provide reliable stock market timing guidance? Do some experts clearly show better intuition about overall market direction than others?

We have accumulated reviews of the public U.S. stock market forecasts of various investing/trading experts for more than two years. With over 4,100 measurements for 50 gurus, including bulls and bears and technicians and fundamentalists, we have critical mass for: (1) assessing the forecasting acumen of the stock market gurus as a group; and, (2) ranking experts according to the accuracy of their past forecasts. This kind of forecasting ability is different from, but may be related to, stock picking expertise.

Note that the overall assessment of the stock market forecasting ability of experts in aggregate is far more reliable, based on sample size and duration, than the evaluations of individuals.

Note also that this study is not a test of whether the outputs of the experts are interesting, stimulating or useful in ways other than predicting the behavior of the overall U.S. stock market.

We update this page and the underlying guru forecast database about weekly.

Snapshot  -  Notes  -  Individual Gurus -  Q&A

SNAPSHOT

We restrict reviews to publicly available material (freely available on the web), putting ourselves in the place of an individual investor trying to locate value in the marketplace, and mindful of concerns about copyright and trade secrets. Sometimes we find public records on the web sites of the experts themselves and sometimes on web sites of other parties (for example, the business media). This approach helps keep a level playing field for reviews, and it allows readers to check easily the context of forecasts and the reasonableness of our judgments.

The following table (last updated 8/23/08) summarizes the results of our reviews of the publicly available forecasts of 50 experts regarding the future direction of the overall U.S. stock market. For each expert, the table shows the total number of measurable forecasts, the numbers of forecasts we judge to be essentially right and essentially wrong, and the accuracy rate. Accuracies range from a low of 12% to a high of 68%. The overall accuracy of the group based on both raw forecast count and on the average of forecaster accuracies (weighting each individual equally) is 48%.

See the Notes below for discussion of the grading process and for cautions on interpreting both the aggregate and the individual expert accuracy rates.

If you believe that past forecasting performance indicates future accuracy (skill, wisdom), you might want to keep tabs on the forecasts of Ken Fisher (and his self-described analysis methodology, "Forecasting (Macro and Micro) and Future Concepts"). If you believe in reversals of fortune, you might follow Bill Fleckenstein. On the third hand, if you expect mean reversion in forecasting (pure luck), you might prefer the "cone of silence". A fourth alternative is to apply Bayesian updating and weight the predictions of experts according to their evolving track records.

In summary, stock market experts as a group do not reliably outguess the market. Some experts, though, may be better than others.

For an unscientifically collected list of reasons why gurus go wrong, see our blog entry of 5/24/07. For additional comic relief, see our blog entries of 8/2/07 and 8/3/07.

NOTES

Please note these points regarding the results in this table:

We selected experts to be graded based on extensive web searches for public archives offering at least marginal stock market forecast sample sizes. (Readers helped identify some commentators.) There may be data availability bias in the aggregation; some types of commentators may be more likely to offer public commentary than others.

Many of the samples for individual experts are small, rendering confidence in the associated accuracies low. The aggregate sample, however, is large.

Assessing market direction forecasts requires judgment (ours) because they sometimes contain ambiguities, equivocations and/or conditional statements. We expect that our judgment errors tend to cancel each other, but we may have biases. Our detailed judgments are available for inspection via the links at the bottom of this page. Further, each individual review provides a link to source commentaries and articles, so that readers may decide whether the essential forecast language properly reflects commentary context.

When an expert commentary is too vague to assess or does not include a stock market forecast, we either do not include it or (if it is of background interest) we include but do not label it either "right" or "wrong." For experts with small samples, we make a few exceptions for commentaries that include no market direction forecast but do offer some other significant and reasonably testable market-related prediction or recommendation.

When an expert issues weekly (monthly) commentaries, we tend to focus on the behavior of the market in the next week (month), unless the forecast specifies some other timeframe.

When judging whether a forecast is correct, we keep in mind empirical benchmarks based on weekly S&P 500 index data for 1950-2006, as listed below. For example, if a guru says investors should be bullish over the next six months (26 weeks), and the market is up by only 2% over that interval, we would judge the call incorrect. Conversely, if a guru says investors should short the market over the next month, and the market is down only 1%, we would again judge the forecast incorrect (a losing position after trading and carrying costs). In summary, the grading process has normalizing or detrending effects such that the aggregate accuracy should probably be around 50%.

• About 50% of all one-week returns are greater than +0.3%.
• About 50% of all four-week returns are greater than +1.0%.
• About 50% of all 13-week returns are greater than +2.4%.
• About 50% of all 26-week returns are greater than +4.5%.
• About 50% of all 52-week returns are greater than +9.5%.

Assessments of different experts cover different timeframes according to the data available. This fact arguably weakens the case for ranking the experts. An expert who is stuck on bullish (bearish) would tend to outperform in a rising (declining) stock market. This effect should eventually cancel for the entire sample across all experts.

The private (for example, paid subscription) forecasts of experts may be more timely and more accurate than the forecasts offered publicly. As noted above, we restrict reviews to publicly available material to: (1) maintain a level playing field for experts reviewed; and, (2) let readers check the reasonableness of our judgments.

With exceptions as noted, these assessments generally address forecasts of overall market direction, not the performance of stock picks. Most stocks, however, exhibit substantial co-movement with the overall market.

For further reading on the performance of experts in general, and what might distinguish a good market forecaster from a bad one, see Blog Synthesis: The Wisdom of Analysts, Experts and Gurus. Consider also also research on how individuals recollect and process advice as summarized in some of the items at especially the third, sixth and seventh items at Blog Synthesis: Individual Investing.

INDIVIDUAL GURUS

The following names link to detailed performance evaluations. Most of these gurus are included in the snapshot above, but some do not fit the methodology and are therefore not comparable.

Some of these individuals have contested our evaluations of their commentaries. In such cases, individual evaluations note the disputes and provide links to details.

Q&A

A reader who is a financial advisor with a large investment services company sent a series of questions/comments regarding Guru Grades. In case others have the same questions, here are the queries and responses...

Question 1: I'm guessing that you get paid by the gurus you hype, no?

Response 1: No. Neither CXO Advisory Group LLC, nor any of its members personally, receive any payments or other compensation from the gurus reviewed on the site. The reviews are attempts to measure the ability of experts to forecast stock market behavior, not "hype."

Question 2: I'm flabbergasted by your top "gurus", as I am familiar with them as well, and they are very poor performers. How can you tout a marketing company (Fisher) as a top money management company? Do you research these firms at all?????

Response 2: The site does not "tout" any companies. It presents reviews of research, some models and some original analyses. The reviews and analyses presented are exactly as described on the site. It is up to readers to decide whether and how this information translates into action for them, or merely flabbergasts them.

Question 3: Why do you compare what people say in articles/periodicals? It means nothing for professionals/investors! Their actual management/performance would certainly be more beneficial to "private investors and financial advisors."

Response 3: As stated above, the material cited addresses the following questions: "Can experts, whether self-proclaimed or endorsed by others (publications), provide reliable stock market timing guidance? Do some experts clearly show better intuition about overall market direction than others?" A reader suggested the concept of comparing multiple experts in this way some time ago. The discussion there further notes that: "This kind of forecasting ability is different from, but may be related to, stock picking expertise." Results are not conclusive that any individual guru has special ability to forecast market behavior. The distribution of the accuracies of individual gurus about the mean is possibly normal. If the reviews of guru stock market forecasting records mean nothing to professionals/investors, then professionals/investors will pay no attention to them. Note that Blog Synthesis: The Wisdom of Analysts, Experts and Gurus catalogs considerable formal research on portfolio performance for different categories of investment managers, including: newsletter writers, hedge fund managers, mutual fund managers and brokerage firm analysts. Guru Grades is just one corner of the site, examining one aspect of investing expertise.

Another reader sent the following suggestion:

Suggestion: While your guru backtesting offers some fun in the way of balloon popping, it does not really offer much in terms of actual investment techniques. I would say the vast majority of your loyal audience is far more interested in [quantitative testing of trading rules] than in testing whether Guru X is or not another snake oil merchant. (Usually he/she is.) I have noticed that your tests of trading rules are widely cited in other places, but I have never seen anybody referencing your popping of another self-appointed guru. Reason? There is no surprise. We do know these guys offer zero. I humbly petition that you stop doing the Guru Grades.

Response: There are two aspects to learning in all fields – what to do, and what not to do. In investing/trading, following the advice of gurus falls mostly into the second category. Many seasoned investors/traders understand that stock market gurus are, by and large, self-promoting and unable to provide accurate financial market forecasts and winning trades with significant reliability. However, other investors are in learning mode regarding guru expertise. Following the advice of experts is an investment technique. Hopefully, Guru Grades is not just fun but also helps these latter investors discover quickly how much attention (and money) they should devote to this technique. Reviews of log files indicate:

Blog entries on quantitative tests of trading rules and academic studies tend to draw a moderate number of readers immediately as posted and then withdraw to a quiet retirement, occasionally picking up a little traffic later via searches or links.

Blog entries on gurus tend to draw more readers initially and have more staying power, especially those covering well-known individuals such as Bob Brinker, Jim Cramer, Ken Fisher, Bill Fleckenstein and the Fast Money experts. Follow-up traffic again comes via searches and links from message boards. If you follow the "Site Hot Spots" feature in the right margin of site pages, you will see that several guru reviews (unlike any specific quantitative analysis or academic study) are consistently among the top 25 site pages in terms of unique visitors each month. This indicates a steady supply of readers seeking information on the credibility of guru pronouncements. The scope of the Guru Grades effort lends support to its conclusions. Nevertheless, the frequency of adding new gurus has dwindled as the number tracked has grown. It is a lot of work to compile a forecast/recommendation history.

Guru Grades is probably not of interest to all segments the CXOadvisory.com audience, but site traffic logs indicate that there is a fairly large segment with continuing interest in this section.

Two readers (one a tracked guru) have suggested that we drop David Nassar from the the list of gurus because: (1) his record is inactive, with no additional forecasts being added; and, (2) his record is relatively short (about 18 months):

Response: As stated, the objectives for Guru Grades are to answer the questions:

"Can experts, whether self-proclaimed or endorsed by others (publications), provide reliable stock market timing guidance?"

"Do some experts clearly show better intuition about overall market direction than others?"

Results indicate that, on average, experts do not provide accurate market forecasts. Dropping gurus from the study as they become inactive might introduce survivorship bias into these results. Several other gurus in the table are at least temporarily absent from the public forecasting forum. A bigger, longer study is probably necessary to determine whether level of accuracy relates to survival.

It is much more difficult to measure whether the dispersion of individual accuracies is random or derived from genuine ability. In either case, it appears that about two-thirds correct may be the best one should expect. David Nassar's sample is relatively short in duration. However, his forecasting style appears to rely on short-term market trends (not broad bull or bear market views), such that there are a fair number of independent forecasts in his sample. In any case, individual reviews take sample size into consideration, and the discussion above includes several cautions regarding the scores of individuals, as follows:

"Note that the overall assessment of the stock market forecasting ability of experts in aggregate is far more reliable, based on sample size and duration, than the evaluations of individuals."

"Many of the samples for individual experts are small, rendering confidence in the associated accuracies low. The aggregate sample, however, is large."

"Assessments of different experts cover different timeframes according to the data available. This fact arguably weakens the case for ranking the experts. An expert who is stuck on bullish (bearish) would tend to outperform in a rising (declining) stock market. This effect should eventually cancel for the entire sample across all experts."