Data snooping bias entails the capture of noise in a dataset that is lucky with respect to a research goal, such as high Sharpe ratio for an investment/trading strategy. Snooping may involve discovery via multiple tests of a lucky subsample in a time series, a lucky parameter value in a model or a lucky alternative model. Small, noisy samples are especially susceptible to snooping. A researcher may inherit snooping bias by using prior biased research as a starting point for further exploration. In any case, snooped research findings degrade or disappear out of sample.
There is an emerging body of research in financial markets based on exploitation of large language model (LLM) capabilities. This research entails prompt engineering, wherein a researcher develops instructions for an LLM to achieve a goal. In presenting research based on LLM outputs, the researcher may describe in detail the sequence of prompts used to elicit these outputs. However, the researcher may previously have tried many variations of these prompts to improve LLM outputs with respect to the research goal. To the degree that LLM “thinking” is opaque, the level of bias derived from this prompt tuning (snooping) is mysterious.
In summary, investors should be skeptical regarding LLM-based research findings due to the potential for prompt snooping.
