A reader requested that we test the stock market forecasts of John Vitale as posted via daily stock market commentary at the Palisades Research web site. The Palisades Research forecasting method involves “a statistical approach to market trading. The technique is unique and proprietary. It relies on two basic elements, ‘money flow’ and ‘investor emotion.’ …The total effort of this program goes into forecasting the direction of the S&P 500 index for the single following day. …Our main program indicates that the next day’s direction can be forecast with a 60% – 70% reliability compared to 53% with buy and hold, but even this is very difficult to maintain in real life.” In this review, we use both regressions and rankings to test the accuracy of the forecasts. Using the record of daily Palisades Research stock market commentary over the period 5/16/06-5/15/08 (499 daily forecasts) and next-day daily opening levels for the S&P 500 index and the Nasdaq 100 index over the same period, we find that…
We infer the daily stock market forecasts from the stated positions for the next trading day, which range across many values from 100% long (completely bullish) to 100% short (completely bearish). There are five trading days in August 2006 for which there are no stated positions for the next trading day. Assuming that positions stated for the next trading day are available after the market close, we calculate daily stock market returns from next-day open to the open on the following trading day.
The following chart superimposes the Palisades Research positions for the next trading day (+1 = 100% bullish and -1 = 100% bearish) on the S&P 500 index over the sample period. While this chart conveys some sense of intervals during which the forward-looking positions are generally more positive or more negative, it is not precise enough to test the accuracy of the positions.
For another perspective, we tabulate the frequencies of forward-looking positions by degree of bullishness/bearishness.
The next chart tabulates the frequencies of Palisades Research daily stated positions for the next trading day. Values greater (less) than zero are bullish/long (bearish/short), while zero itself means neutral/out of stocks. Over the entire sample period, the forward-looking positions are bullish, neutral and bearish 37.3%, 31.5% and 31.3% of the time, respectively.
As a first test of forward-looking position accuracy, we use a regression.
The following scatter plot relates next-day change in the S&P 500 index to the daily forward-looking position by Palisades Research across the entire sample period. If the positions are predictive (if greater bullishness indicates increasingly positive returns and greater bearishness indicates increasingly negative returns), a best-fit line should run from the lower left to the upper right of the chart. However, the best-fit line is flat and lies on the horizontal axis. The Pearson correlation between the two series is 0.00, and the R-squared statistic is 0.000, indicating that the forward-looking positions have no predictive power for next-day stock returns.
A similar test using the Nasdaq 100 index instead of the S&P 500 index yields a Pearson correlation of 0.01 an R-squared statistic of 0.000, again indicating no predictive power for the stated positions for the next trading day.
As a second test of forward-looking position accuracy, we calculate average daily returns by range of stated bullishness/bearishness.
The next chart summarizes average next-day returns for the S&P 500 index for various ranges of Palisades Research forward-looking positions over the entire sample period, from extreme bearishness on the left to extreme bullishness on the right. The average daily return for all days in the sample period is 0.02%. If the positions are reliably predictive, the average daily returns should increase regularly from left to right. There is no such progression. In fact, there is no discernible regularity in average daily returns. The highest average next-day return occurs after moderately bearish forward-looking positions. The lowest average next-day average return occurs after neutral positions. Again, there is no evidence of predictive power in the forward-looking positions.
As a third and most simple test of forward-looking position accuracy, we count next-day advances and declines for bullish, neutral and bearish positions.
The following table shows the percentages of next-day S&P 500 index returns that are positive and negative according to whether the associated Palisades Research forward-looking position is to some degree bullish, neutral or to some degree bearish over the entire sample period. Across the entire sample, 54.0% (45.8%) of trading days have positive (negative) returns. Results associated with positions do not approach the potential 60%-70% reliability cited above. In fact, the 45.5% accuracy rate for bearish positions is notably unreliable.
For the subset of daily forward-looking positions that is 100% bullish (bearish), next-day S&P 500 index returns are positive 57.6% (56.3%) of the time. In other words, there is hardly any difference between fully bullish and fully bearish positions with respect to the average direction of the stock market the next day.
In summary, testing indicates that the daily stated positions for the next trading day from Palisades Research have no power to predict next-day stock market returns.
This review tests the general market timing value of the daily forward-looking of Palisades Research and not their stated investing performance.
Comments from Palisades Research founder John Vitale:
You have misconstrued what we do and how we do it. I trade Rydex funds at the close for the next day. Blindly using my signals to trade the NDX or SPX the next morning and closing out the following morning is never suggested. Considering slippage and commission it would most likely lead to a loss. In fact I ask people not to use my signals. They are there so potential investors can check our results. Our record stands as stated on my site. As a legitimate registered investment advisor in California I can not do otherwise. Please provide a telephone number and email address so that I can send you the past 28 months of actual trading. This is what you should have asked for prior to writing your article. If you wish to be taken seriously you must be accurate and also not hide behind a Web site with limited contact information. I think what you are trying to do is good, but please do it correctly.
Responses:
The above review is a test of the predictive power of the Palisades Research daily stated positions for the next trading day . The review explicitly states: “This review tests the general market timing value of the daily forward-looking positions of Palisades Research and not their claimed investing performance.”
The description of the Palisades Research timing method states: “The total effort of this program goes into forecasting the direction of the S&P 500 index for the single following day.” The above review therefore uses the S&P 500 index for testing the next-day predictive power of forward-looking positions. The review is not a test of trading returns.
The daily commentaries are not time-stamped, so the historical record does not show whether the posted forward-looking positions are available at the market close. Re-running the tests using same-day close to next-day close for the S&P 500 index rather than next-day open to following-day open yields nearly identical results, as follows:
For the regression test, the Pearson correlation changes from 0.00 to -0.01, and the value of the R-squared statistic remains 0.000. These alternative results indicate that the stated positions for the next trading day have no predictive power for the behavior of S&P 500 index the that day.
The following chart shows the alternative average next-day returns for the S&P 500 index for various ranges of Palisades Research forward-looking positions. Results are a little different from the version above, but they still do not demonstrate systematic predictive power.
The following table shows the alternative percentages of next-day S&P 500 index returns that are positive and negative according to whether the associated Palisades Research forward-looking position is to some degree bullish, neutral or to some degree bearish. These alternative percentages vary only slightly from those shown above. Results again do not approach the potential 60%-70% reliability cited above, and the accuracy rate for bearish positions is still notably unreliable.
For linking purposes, please specify where on the Palisades Research site you ask people not to use your signals.
Reviews at CXOadvisory.com use only data readily available to all investors (posted on the web), so that readers can replicate or extend any analyses presented. If you post your actual trading records so that anyone can review and analyze them, we will at a minimum link to the posting.
Please identify any inaccuracies in the review as a test of the predictive power of your daily forward-looking positions so that we can resolve them.
The information presented at CXOadvisory.com is extremely transparent with regard to sources and methods so that readers can check the work and do their own additional analyses as desired. Whether readers take the information seriously is up to them.
It is easy to contact us directly or via webform, as indicated at the bottom of the home page and on the About and Contact Us pages. We respond to every civil message that includes a valid reply-to email address as soon as practicable.
Additional itemized comments from Palisades Research founder John Vitale on 5/26/08, with interleaved responses:
Comment 1. Nowhere on my site is the word recommend or recommendation stated or implied. For you to say that I am recommending a certain trade when what I am doing is clearly posting my closing position for the day is (according to my lawyer) libelous. The definition of the word forecast is not the same as the definition of recommend. And I sincerely urge you to reflect this on your site. As a registered investment advisor I charge for my recommendations and they are not published publicly.
Response 1: The Palisades Research daily commentaries, in the bolded lead summary statements, generally describe positions as forward-looking, applicable to the next trading day. For example (underlining added to quotes):
From the commentary for Friday, 5/23/08: “Current position: 60% Short, Rydex Inverse Nasdaq 100 2x, 40% money market for Tuesday.”
From the commentary for Thursday, 5/22/08: “Current position: 80% Short, Rydex Inverse Nasdaq 100 2x, 20% money market for Friday.”
From the commentary for Wednesday, 5/21/08: “Current position: 20% Long, Rydex Nasdaq 100 2x, 80 % money market for Thursday.”
From the commentary for Tuesday, 5/20/08: “Current position: 25% Short, Rydex Inverse Nasdaq 100 2x, 75% money market for Wednesday.”
From the commentary for Monday, 5/19/08: “Current position: 100% money market for Tuesday.”
For literal accuracy, we have substituted descriptions such as “stated position for the next trading day” or “forward-looking position” for “recommended next-day position” throughout the analysis.
Comment 2. I did not state that my returns as shown from Jan 1, 2006 through March 23, 2008 reflect a 60% to 70% accuracy. I said “the basic program indicates that the next day’s direction can be forecast with a 60% to 70% reliability compared to 53% with buy and hold, but even this is difficult to maintain in real life.” The “basic program” refers to the computer output, “real life” refers to actual trading. Things happen in real life. There are power failures, unforeseen events like 911 and a wide range of things that can prevent this type of continuous return. However, we are running more than +62% accurate since mid November 2007, over seven months, and this is “real life”.
Response 2: The above analyses do not attempt to measure the accuracy of Palisades Research returns, however “accuracy” might be defined for returns. The analyses instead test the power of the daily Palisades Research forward-looking position to predict the change in the S&P 500 index the next trading day over the two-year period from 5/16/06 through 5/15/08. The results of the analyses do not support a belief that the daily positions have predictive power.
Comment 3. If you want to tell people not to follow my signals because they can not trade the same investment vehicles that I trade for my clients at the time I trade them, then that is good advice. I tell them not to trade the signals right on the home page of my web site. For you to do a statistical analysis on something not remotely similar to what I am doing, then claiming it is related to my recommendations when I make no recommendations, is irresponsible if not dishonest. You referred to my performance figures as claimed investing performance, the word “claimed” implies that our performance figures might not be true. …A more fair approach for you to take would be to say “stated investment performance”. I expect a correction.
Response 3: See the previous response. We leave it to readers to decide how they should invest and trade. If the daily forward-looking positions in the Palisades Research commentaries are “not remotely similar” to what you are doing, why bother to post them? To address the sensitivity indicated in the comment, we have changed “claimed investment performance” to “stated investment performance.”
Comment 4. I am not going to comment on your Pearson correlation because it does not pertain to anything on my site. But I suggest you use a Monte Carlo simulation instead of Pearson because it will give you more “real world” information.
Response 4: Regressions (correlation and coefficient of determination) and sorts are generally accepted methods of investigating relationships between pairs of financial variables. These concepts do pertain to the relationship between the daily Palisades Research forward-looking positions and their implied predictions for the next-day change in the S&P 500 index. For discussion of the advantages and disadvantages of regressions and sorts, see the June 2007 draft paper entitled “Dissecting Anomalies” by Eugene Fama and Kenneth French. Monte Carlo methods generally apply to approximation of solutions for mathematically intractable problems involving multiple input variables, with each variable involving some level of uncertainty. In the above tests, the two input variables (the daily Palisades Research forward-looking position and the change in the S&P 500 index the next trading day) are definitive and readily subject to analysis via regression and sorts.
Comment 5. Registered investment advisors are highly regulated, especially California Registered Investment Advisors. You will find that few are willing to risk the large fines and prison terms that are the consequences of misrepresentation. News-letter publishers are much less regulated and more problems are found there. If you hunt you will also find web sites that are completely unregulated but have an agenda. A good rule of thumb is first look to find a legitimate, name, address and phone number as well as actual email. If those things are missing you have found someone with something to hide and you need to go no further.
Response 5: We leave it to readers to set their own standards for the legitimacy of investing and trading advice.
