Innumeracy and Look-ahead Bias in LLMs?

Recent research in accounting and finance finds that large language models (LLM) beat humans on a variety of related tasks, but the black box nature of LLMs obscures why. Is LLM outperformance real? In his December 2024 paper entitled “Caution Ahead: Numerical Reasoning and Look-ahead Bias in AI Models”, Bradford Levy conducts a series of experiments to open the LLM black box and determine why LLMs appear to perform so well on accounting and finance-related tasks. He focuses on numerical reasoning and look-ahead bias. Based on results of these experiments, he finds that:

(more…)