This study aims to evaluate the performance of zero-shot large language models (LLMs) against traditional supervised machine learning models for next-day market direction prediction using the S&P 500 index. Three LLMs (GPT-4o-mini, Claude Haiku 4.5 and Qwen3.5) were compared against two supervised ML models (Logistic Regression and Random Forest) using the same test dataset of 199 rows of S&P 500 features. These were compared against a majority-class baseline of 55.78%, with the supervised ML models being trained on 792 rows of past data.

It was found that no model produced statistically significant results against the baseline, with the best performing model being Random Forest with an accuracy of 56.78% and the only model which gained a positive Cohen’s Kappa score of 0.1135. Yet even this was not statistically significant against the baseline, failing to reject the null hypothesis. Additionally, it was found that all three LLMs underperformed the majority-class baseline with majority-vote accuracies between 46.23% and 48.24% with negative Cohen’s Kappa scores.

A key finding in this study is that the LLMs are not just randomly wrong, but are consistently wrong, as seen through their high Fleiss’ Kappa scores where GPT-4o-mini and Claude Haiku 4.5 were put into the band of “almost perfect” and Qwen3.5 achieving “fair”. This points towards a downward systematic bias in the LLMs’ predictions which could be attributed to the skew of the training corpus towards market crashes and financial crises, a risk-averse bias, or the LLMs’ struggle to reason about numerical data.