Extracting Sentiment Probabilities from LLMs

Generative large language models (LLM), such as ChatGPT, are best known for conversational summation of complex information. Their use in financial forecasting focuses on discrete news sentiment signals of positive (1), neutral (0) or negative (-1). Is there a way to extract more granularity in LLM sentiment estimates? In their October 2024 paper entitled “Cut the Chit-Chat: A New Framework for the Application of Generative Language Models for Portfolio Construction”, Francesco Fabozzi and Ionut Florescu present Logit Extraction as a way to replace discrete LLM sentiment labels with continuous sentiment probabilities and apply results to ranking stocks for portfolio construction. Logit Extraction exploits the inner workings of LLMs to quantify sentiment strength. They test it on four LLMs: Mistral, Llama, ChatGpT-3.5 and ChatGPT-4. Their benchmark model is the specialized, quantitative FinBERT. They compare the abilities of each LLM to those of FinBERT in replicating human-assigned sentiment labels and generating long-short portfolio risk-adjusted returns, with and without Logit Extraction. Inputs are initial-release headlines from news alerts covering a single company published from 30-minutes before market open on the previous day to 30-minutes before market open on the day of trading during January 2010 through October 2020. They aggregate headlines published on non-trading days for long-short trading the next trading day. Portfolio trades occur at the open each trading day and are limited to stocks in the news the day before (an average of 46). Using the specified 216,837 news headlines and associated daily returns across 263 unique firms, they find that: (more…)