Imagined Markets, Imagined Research Findings?

Experimental (researcher-imagined) asset markets provide a controlled environment for testing hypotheses about investor behaviors. Do limits on abilities of researchers to model markets realistically, and researcher incentives/motivations, jeopardize the credibility of associated studies? In their December 2024 paper entitled “Do Experimental Asset Market Results Replicate? High-powered Preregistered Replications of 17 Claims”, Christoph Huber, Felix Holzmeister, Magnus Johannesson, Christian König-Kersting, Anna Dreber, Juergen Huber and Michael Kirchler attempt to replicate findings on behavioral drivers of asset prices from four prominent studies based on experimental asset markets.  Specifically, they seek to replicate 17 published findings on: (1) associations between asset prices and emotions, self-control, experience and gender; and, (2) trader characteristics (cognitive reflection, fluid intelligence and theory of mind) that explain trading success. 14 of the 17 published findings are reported as statistically significant. To mitigate snooping bias, they fully document (preregister) study design, analyses and statistical tests before collecting any data. When evaluating replicability, they consider statistical significance (confirmation of prior finding with p < 0.05) and the ratio of effect size in the replication to that in the original study. Using new data from 166 experimental asset markets with 1,544 participants (more than used in the original studies), they find that:

(more…)