Uncovering Language Disparity of ChatGPT on Retinal Vascular Disease Classification: Cross-Sectional Study

Abstract

Background Benefiting from rich knowledge and the exceptional ability to understand text, large language models like ChatGPT have shown great potential in English clinical environments. However, the performance of ChatGPT in non-English clinical settings, as well as its reasoning, have not been explored in depth. Objective This study aimed to evaluate ChatGPT's diagnostic performance and inference abilities for retinal vascular diseases in a non-English clinical environment. Methods In this cross-sectional study, we collected 1226 fundus fluorescein angiography reports and corresponding diagnoses written in Chinese and tested ChatGPT with 4 prompting strategies (direct diagnosis or diagnosis with a step-by-step reasoning process and in Chinese or English). Results Compared with ChatGPT using Chinese prompts for direct diagnosis that achieved an F 1 -score of 70.47%, ChatGPT using English prompts for direct diagnosis achieved the best diagnostic performance (80.05%), which was inferior to ophthalmologists (89.35%) but close to ophthalmologist interns (82.69%). As for its inference abilities, although ChatGPT can derive a reasoning process with a low error rate (0.4 per report) for both Chinese and English prompts, ophthalmologists identified that the latter brought more reasoning steps with less incompleteness (44.31%), misinformation (1.96%), and hallucinations (0.59%) (all P 1 -score (P=.04) between Chinese and English prompts. In short, when prompted in English, ChatGPT exhibited enhanced diagnostic and inference capabilities for retinal vascular disease classification based on Chinese fundus fluorescein angiography reports. Conclusions ChatGPT can serve as a helpful medical assistant to provide diagnosis in non-English clinical environments, but there are still performance gaps, language disparities, and errors compared to professionals, which demonstrate the potential limitations and the need to continually explore more robust large language models in ophthalmology practice.

Contribution highlights

• Objective This study aimed to evaluate ChatGPT's diagnostic performance and inference abilities for retinal vascular diseases in a non-English clinical environment.

Extracted from the public abstract; no claims have been added.

Code and data

Publication metadata

DOI

10.2196/51926

PMID

38252483

PMCID

PMC10845019

Volume

26

Pages / article

e51926

Version relationship

No separate preprint-to-publication relationship is currently recorded.

BibTeX

@article{Liu2024Uncoveringlanguage,
  title = {Uncovering Language Disparity of ChatGPT on Retinal Vascular Disease Classification: Cross-Sectional Study},
  author = {Xiaocong Liu and Jiageng Wu and An Shao and Wenyue Shen and Panpan Ye and Yao Wang and Juan Ye and Kai Jin and Jie Yang},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  url = {https://www.jmir.org/2024/1/e51926},
  doi = {10.2196/51926},
  pmid = {38252483},
  volume = {26},
  pages = {e51926}
}