How well does the Morningstar fund rating system (one star to five stars) work? In their November 2016 paper entitled “The Morningstar Rating for Funds: Analyzing the Performance of the Star Rating Globally”, suggested for review by a subscriber, Jeffrey Ptak, Lee Davidson, Christopher Douglas and Alex Zhao analyze the global performance of star ratings in terms of ability to predict fund performance. They use two test methodologies:
- Monthly two-stage regressions that test the ability of fund star ratings to add value to a linear factor model for each asset class at a one-month horizon. The first stage estimates fund dependencies (betas) on commonly used predictive factors over the past 36 months. The second stage measures the ability of fund star ratings to add predictive power to those betas in the following month. For stock funds, they consider fee, equity market, size, value and momentum factors. For bond funds, they consider fee, credit and term factors. For stock-bond funds, they consider all these factors. For alternative asset class funds, they consider fee and equity market factors.
- An event study that tracks performances of equally weighted portfolios of funds formed by prior-month star rating over the next 1, 3, 6, 12, 36 and 60 months.
Using fund categories, monthly fund star ratings and returns, and asset class factor returns during January 2003 through December 2015, they find that:
- From regression tests (see the table below):
- For stock funds, star ratings exhibit some but not conclusive incremental predictive power. The monthly premium of 5-star funds over 1-star funds is 0.09% (1.03% annualized).
- For bond funds, star ratings exhibit significant incremental forecasting ability. The monthly premium of 5-star funds over 1-star funds is 0.09% (1.09% annualized).
- For stock-bond funds, star ratings exhibit significant incremental forecasting ability. The monthly premium of 5-star funds over 1-star funds is 0.15% (1.75% annualized).
- For alternatives funds, star ratings do not exhibit reliable incremental forecasting ability. The monthly premium of 5-star funds over 1-star funds is 0.04% (0.48% annualized).
- Regarding the event study (see the chart below):
- For stock funds at a three-year horizon, 5-star funds on average outperform 1-star funds by an annualized 0.64%.
- For bond funds at a three-year horizon, 5-star funds on average outperform 1-star funds by an annualized 0.35%.
- For stock-bond funds at a three-year horizon, 5-star funds on average outperform 1-star funds by an annualized 0.59%.
- For alternative funds at a three-year horizon, 5-star funds on average outperform 1-star funds by an annualized 0.17%.
- Persistence of the performance gap between high-rated and low-rated funds likely derives from the strong inverse correlation between star ratings and fees.
The following table, extracted from the paper, summarizes annualized differences in premiums (percentages) between 3-star funds and each of 1-star, 2-star, 4-star and 5-star funds by category as generated by the regression methodology. For equity, bond (fixed income) and stock-bond (balanced) funds, 1-star and 2-star funds consistently underperform 3-star funds, and 4-star and 5-star funds consistently outperform 3-star funds, and results are systematic across star ratings. For alternative funds, however, results are not systematic and 1-star and 2-star funds outperform 3-star funds (3-star funds are the worst performers).
The following chart, taken from the paper, summarizes event study performance by star rating across all fund categories. Higher-star funds systematically and consistently outperform lower-star funds, but the performance differences are small.
In summary, evidence indicates that the Morningstar star rating system has some ability to predict risk-adjusted returns in the short term and does more good than harm to investors over the long term.
Cautions regarding findings include:
- Per the December 2013 “Morningstar Methodology Paper”, the last “significant” changes to the star rating methodology occurred in 2006, more than three years after the beginning of the sample period. That paper further states: “In addition to the [significant] changes, Morningstar has occasionally added new categories to the category structure in each market and those changes can also impact the fund ratings.” One of the authors of the paper reports that post-2003 methodology changes are not material to the study.
- As noted in the paper, the regression methodology is short-term (one month ahead) and may not reflect long-term investor experience.
- As noted in the paper, the event study does not control for fund risks.
- Reliably exploiting findings requires (impractically) holding a portfolio of many mutual funds with periodic reformation.
- Applying the test methodologies to multiple fund subsamples (categories) introduces snooping bias, such that the strongest findings overstate expectations.
