Evaluation of artificial intelligence-generated layperson’s summaries from abstracts of vascular surgical scientific papers

Background The study aimed to assess the efficacy of ChatGPT 3.5, an artificial intelligence (AI) language model, in generating readable and accurate layperson’s summaries from abstracts of vascular surgery studies. Materials and methods Abstracts from four leading vascular surgery journals published between October 2023 and December 2023 were utilized. A ChatGPT prompt for developing layperson’s summaries was designed based on established methodology. Readability measures and grade-level assessments (RR-GLIs) were compared between original abstracts and ChatGPT-generated summaries. Two vascular surgeons evaluated a randomized sample of ChatGPT summaries for clarity and correctness. Readability scores of original abstracts were compared with ChatGPT-generated layperson’s summaries using a t-test. Moreover, a sub-analysis based on abstract topics was performed. Cohen’s kappa assessed interrater reliability for accuracy and clarity. Results One-hundred and fifty papers were included in the database. Statistically significant differences were observed in RR-GLIs between original abstracts and AI-generated summaries, indicating improved readability in the latter (mean Global Readability Score of 36.6±13.8 in the original abstract and of 50.5±11.1 in the AI-generated summary, p<0.001). This trend persisted across abstract topics and journals. While one physician found all summaries correct, the other noted inaccuracies in 32% of cases, with mean rating scores of 4 and 4.7, respectively, and no inter-observer agreement (k value=-0.1). Conclusions ChatGPT demonstrates utility in producing patient-friendly summaries from scientific abstracts in vascular surgery, although the accuracy and quality of AI-generated summaries warrant further scrutiny.