AI CFAs?

Can large language models (LLM) such as ChatGPT and GPT-4 pass the Chartered Financial Analyst (CFA) exam, which covers fundamentals of investment tools, asset valuation, portfolio management and wealth planning? In their October 2023 paper entitled “Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on Mock CFA Exams”, Ethan Callanan, Amarachi Mbakwe, Antony Papadimitriou, Yulong Pei, Mathieu Sibue, Xiaodan Zhu, Zhiqiang Ma, Xiaomo Liu and Sameena Shah investigate whether ChatGPT and GPT-4 could pass the CFA exam. They ask the models to respond to mock exam questions from the first two of the three levels on the exam:

• Level I – 180 standalone multiple choice questions (using questions from five mock exams).
• Level II – 22 vignettes and 88 accompanying multiple choice questions, with a higher proportion requiring interpretation of numerical data and calculations than found in Level I (using questions from two mock exams).
• Level III – a mix of vignette-related essay questions and vignette-related multiple choice questions (untested due to the difficulty of assessing essay responses).

They assess responses to the Level I and II mock exam questions via three approaches:

1. Gauging inherent model reasoning abilities without providing any correct examples.
2. Facilitating model acquisition of new knowledge by providing examples of good responses for either (a) a random sample of questions within level or (b) one question from each exam topic.
3. Prompting the models to address each question step-by-step and to show their work for calculations.

They then compared responses of the two models to approved answers and estimate whether either could pass based on proficiency thresholds reported by CFA exam takers on Reddit. Using mock CFA Level I and II exam questions and the three test approaches as described above, they find that: (more…)