Why are expert economic and financial (econometric) forecasters so inaccurate? In his April 2019 presentation package for a graduate course at Cornell entitled “The 7 Reasons Most Econometric Investments Fail”, Marcos Lopez de Prado enumerates shortcomings of standard econometric statistical methods, which concentrate on multivariate linear regressions. In contrast, advanced computational methods that exploit machine learning are ascendant in many other scientific fields, because they avoid (likely unrealistic) assumptions regarding actual data generation (such as linearity). Based on reviews of econometric texts and the body of econometric research, he concludes that:
- Standard econometric methods achieve time series stationarity (consistency over time) by erasing time series memory (information from the path followed by data). Machine learning methods are more flexible in exploiting path information.
- Standard econometric methods do not address noise and bias in correlations (and betas) and hence have difficulty distinguishing between signal and noise. Cross-sectional studies are especially sensitive to disruption by outliers, activation thresholds and regime changes.
- Standard econometrics methods are adapted from biology, for which out-of-sample forecasting is not critical, leading to spurious claims of causation and false investment strategies.
- Standard econometric methods require researchers to specify both the predictive variables and the algorithm (function, or formula) that relate predictive and predicted variables. Given the complexity of financial systems, this joint requirement is unrealistic. In contrast, machine learning discovers the best algorithm for a given set of predictive variables.
- Standard econometric methods employ p-values at conventional levels to decide statistical significance of findings. In practice, this approach impounds considerable data snooping bias from multiple testing within and across studies, and markets adapt to new findings and depress p-values out-of-sample. Machine learning accommodates the superior Mean Decrease Accuracy method, which: a) discovers the best algorithm for a group of predictive variables in the training dataset, and estimates test dataset (out-of-sample) accuracy; b) shuffles values of one predictive variable at a time and re-estimates out-of-sample accuracy; and, c) assesses decays in out-of-sample accuracy from shuffling values of each variable.
- Standard econometric methods are mostly silent about repeating tests on a noisy training dataset to select among variations in an algorithm (such as parameter values). Such testing makes noise (luck) important in findings for the selected algorithm. Machine learning offers ways to suppress and quantify such overfitting.
- Standard econometric methods are also mostly silent about repeating tests on a noisy test dataset (out-of-sample). Again, machine learning offers ways to suppress and quantify such overfitting.
In summary, academics and investment professionals continue to use research methods ill-suited to the complexity of economies and markets, largely ignoring the superiority of new machine learning methods in these environments.
Cautions regarding conclusions include:
- Machine learning methods are out-of-reach for many investors, who would bear fees for delegating to the (rare) fund managers who understand and use them as described.
- Moreover, investors not taking advantage of such methods may be at material disadvantage in financial markets versus those who do.
For additional presentations related to the same graduate course, see “Investment Strategy Development Coursework”.
