Article Dans Une Revue European Geriatric Medicine Année : 2025

Evaluating how different versions of ChatGPT align with expert opinions on geriatric script concordance tests

Résumé

Purpose: Large Language Models (LLMs), such as ChatGPT, are increasingly used in medical education, particularly for training clinical reasoning. Script Concordance Tests (SCTs) are valuable tools for assessing reasoning under uncertainty. Our objective was to evaluate the performance of four ChatGPT versions (3.5, 4, 4o, and 5) on SCTs compared to Geriatric Medicine experts.Methods: Twenty validated SCT questions covering diverse geriatric topics were administered to ChatGPT-3.5, 4, 4o, and 5, as well as a panel of 43 experts. SCTs assess responses on a five-point Likert scale, and scores were calculated using aggregate expert answers as the reference. Performance was analyzed based on alignment with expert majority responses.Results: ChatGPT-3.5 achieved a score of 8/20, aligning with the majority of experts on 15% of responses. ChatGPT-4 and -5 demonstrated significantly improved performance, scoring 15.6/20 and 15.2/20, respectively, and aligning with the majority on 45% of responses. ChatGPT 4o scored 13.6/20, slightly underperforming GPT-4, potentially due to differences in its architecture or training data. While GPT-4 and 5 excelled in reasoning and coherence, GPT-3.5 struggled with nuanced clinical scenarios.Conclusion: ChatGPT-4 and -5 performed comparably to geriatric experts in SCT-based reasoning, highlighting its potential for medical education. It can simulate case discussions, enhance autonomous learning, and support formative assessments. However, limitations such as variability across versions and question-specific challenges emphasize the need for further evaluation of LLMs across broader clinical domains. These findings support the integration of advanced LLMs into medical education to complement traditional teaching methods.

Fichier non déposé

Dates et versions

inserm-05387147 , version 1 (28-11-2025)

Identifiants

Citer

Matthieu Lilamand, Théodore Decaix, Pierre-Antoine Gourraud, Richard Dufour, Gaëtan Gavazzi, et al.. Evaluating how different versions of ChatGPT align with expert opinions on geriatric script concordance tests. European Geriatric Medicine, 2025, Online ahead of print. ⟨10.1007/s41999-025-01334-5⟩. ⟨inserm-05387147⟩
82 Consultations
0 Téléchargements

Altmetric

Partager

  • More