What happens to the rankings of Guru Grades after weighting each forecast by forecast horizon and specificity? In their March 2017 paper entitled “Evaluation and Ranking of Market Forecasters”, David Bailey, Jonathan Borwein, Amir Salehipour and Marcos Lopez de Prado re-evaluate and re-rank market forecasters covered in Guru Grades after weighting each forecast by these two parameters. They employ original Guru Grades forecast data as the sample of forecasts, including assessments of the accuracy of each forecast. However, rather than weighting each forecast equally, they:
- Apply to each forecast a weight of 0.25, 0.50, 0.75 or 1.00 according to whether the forecast horizon is less than a month/indeterminate, 1-3 months, 3-9 months or greater than 9 months, respectively.
- Apply to each forecast a weight of either 0.5 for less specificity or 1.0 for more specificity.
Using a sample of 6,627 U.S. stock market forecasts by 68 forecasters from CXO Advisory Group LLC, they find that:
- Applying the two weights specified above increases the accuracy of 38% of forecasters and decreases the accuracy of 62% (see the chart below).
- In both the original Guru Grades study and this study, about 40% of forecasters have an accuracy between 40% and 50%. This study produces more extreme accuracies, with 3% (6%) of forecasters having accuracies between 10% and 20% (70% and 80%).
- The average absolute change in ranking of forecasters between the original Guru Grades study and this study is 4.6. In other words, the average guru moved up or down 4.6 places in ranking by accuracy from 1 to 68.
The following chart, taken from the paper, compares accuracies of forecasters from this study and from the original Guru Grades study. More forecasters lose than gain accuracy in this study compared to the Guru Grades study.
In summary, weighting the forecasts from Guru Grades by forecast horizon and specificity mostly lowers forecaster accuracies and moderately shifts rankings.
Cautions regarding findings include:
- The original Guru Grades study cites 6,582 graded forecasts rather than 6,627 as above. The source data confirms 6,582. There are discrepancies for four forecasters [reported to the contact author].
- Assigning horizon and specificity weights to forecasts involves some judgment and/or approximation.
- Most of the cautions of the original Guru Grades study apply to this study.
