AarXiv13 Aug 2025Cited 1×

Performance of GPT-5 Frontier Models in Ophthalmology Question Answering

Antaki, Fares, Mikhail, David, Milad, Daniel, Mammo, Danny A, Sharma, Sumit, Srivastava, Sunil K, Chen, Bing Yu, Touma, Samir, Sevgi, Mertcan, El-Khoury, Jonathan, Keane, Pearse A, Chen, Qingyu, Tham, Yih Chung, Duval, Renaud

Abstract

Fares Antaki, MDCM1,2,3,4 ; David Mikhail 5 , BSc; Daniel Milad, MD 2,3,6 ; Danny A Mammo, MD 1 ; Sumit Sharma, MD 1,7 ; Sunil K Srivastava, MD 1 ; Bing Yu Chen, MDCM 8 ; Samir Touma, MDCM2,3,6 ; Mertcan Sevgi, MD 9,10 ; Jonathan El-Khoury, MD 2,3,6 ; Pearse A Keane, MD 9,10 ; Qingyu Chen, PhD 11 ; Yih Chung Tham, PhD 12,13,14 ; Renaud Duval, MD 2,6

A figure from Performance of GPT-5 Frontier Models in Ophthalmology Question Answering — fig. from the paper

§ The Valyu brief

The paper, in five minutes

Read from the full text via Valyu Contents

GPT-5-high reached 96.5% accuracy on a 260‑question ophthalmology test, beating most rival models and revealing a low‑cost, high‑performance GPT‑5-mini‑low option.

What’s new

i.Evaluated 12 GPT‑5 configurations plus o1‑high, o3‑high, and GPT‑4o on a 260‑question ophthalmology dataset
ii.Applied a Bradley‑Terry model for head‑to‑head ranking of model accuracy
iii.Developed a reference‑anchored, pairwise LLM‑as‑a‑judge framework to assess rationale quality
iv.Performed token‑based cost‑accuracy analysis to identify Pareto‑optimal configurations
v.Introduced an autograder framework for scalable evaluation of LLM answers in ophthalmology

How they did it

The authors ran each model on 260 closed‑access multiple‑choice questions from the American Academy of Ophthalmology BCSC, measuring raw accuracy as the primary outcome. They further ranked models using a Bradley‑Terry model, judged answer rationales with an LLM‑as‑judge system, and estimated token‑based costs to analyze accuracy‑cost trade‑offs.

What they found

GPT‑5‑high achieved the highest accuracy of 0.965 (95% CI 0.942‑0.985)
GPT‑5‑high outperformed all GPT‑5‑nano variants (P < .001), o1‑high (P = .04), and GPT‑4o (P < .001)
GPT‑5‑high was not significantly better than o3‑high, which scored 0.958 (95% CI 0.931‑0.981)
GPT‑5‑high ranked first in accuracy (1.66× stronger than o3‑high) and rationale quality (1.11× stronger than o3‑high)
GPT‑5‑mini‑low was identified as the most favorable low‑cost, high‑performance configuration on the Pareto frontier

§ The Valyu brief

Reading the full paper and taking notes. This takes a few seconds…

§ Ask this paper

Ask a question about this paper

Valyu reads the full text and answers from what the paper actually says.

§ Related work in other archives

Performance of GPT-5 Frontier Models in Ophthalmology Question Answering

The paper, in five minutes

Ask a question about this paper

Similar papers in all four archives

The paper, in five minutes

Similar papers in all four archives