Background The study aimed to assess the efficacy of ChatGPT 3.5, an artificial intelligence (AI) language model, in generating readable and accurate layperson’s summaries from abstracts of vascular surgery studies. Materials and methods Abstracts from four leading vascular surgery journals published between October 2023 and December 2023 were utilized. A ChatGPT prompt for developing layperson’s summaries was designed based on established methodology. Readability measures and grade-level assessments (RR-GLIs) were compared between original abstracts and ChatGPT-generated summaries. Two vascular surgeons evaluated a randomized sample of ChatGPT summaries for clarity and correctness. Readability scores of original abstracts were compared with ChatGPT-generated layperson’s summaries using a t-test. Moreover, a sub-analysis based on abstract topics was performed. Cohen’s kappa assessed interrater reliability for accuracy and clarity. Results One-hundred and fifty papers were included in the database. Statistically significant differences were observed in RR-GLIs between original abstracts and AI-generated summaries, indicating improved readability in the latter (mean Global Readability Score of 36.6±13.8 in the original abstract and of 50.5±11.1 in the AI-generated summary, p<0.001). This trend persisted across abstract topics and journals. While one physician found all summaries correct, the other noted inaccuracies in 32% of cases, with mean rating scores of 4 and 4.7, respectively, and no inter-observer agreement (k value=-0.1). Conclusions ChatGPT demonstrates utility in producing patient-friendly summaries from scientific abstracts in vascular surgery, although the accuracy and quality of AI-generated summaries warrant further scrutiny.

Evaluation of artificial intelligence-generated layperson’s summaries from abstracts of vascular surgical scientific papers / Walter Dorigo, Elena Giacomelli, Cristiano Calvagna*, Filippo Griselli*, Sara Speziali, Aaron Thomas Fargion, Sandro Lepidi**, Raffaele Pulli, Mario D’Oria**. - In: JVS-VASCULAR INSIGHTS. - ISSN 2949-9127. - ELETTRONICO. - (In corso di stampa), pp. 0-0.

Evaluation of artificial intelligence-generated layperson’s summaries from abstracts of vascular surgical scientific papers

Walter Dorigo;Elena Giacomelli;Sara Speziali;Aaron Thomas Fargion;Raffaele Pulli;
In corso di stampa

Abstract

Background The study aimed to assess the efficacy of ChatGPT 3.5, an artificial intelligence (AI) language model, in generating readable and accurate layperson’s summaries from abstracts of vascular surgery studies. Materials and methods Abstracts from four leading vascular surgery journals published between October 2023 and December 2023 were utilized. A ChatGPT prompt for developing layperson’s summaries was designed based on established methodology. Readability measures and grade-level assessments (RR-GLIs) were compared between original abstracts and ChatGPT-generated summaries. Two vascular surgeons evaluated a randomized sample of ChatGPT summaries for clarity and correctness. Readability scores of original abstracts were compared with ChatGPT-generated layperson’s summaries using a t-test. Moreover, a sub-analysis based on abstract topics was performed. Cohen’s kappa assessed interrater reliability for accuracy and clarity. Results One-hundred and fifty papers were included in the database. Statistically significant differences were observed in RR-GLIs between original abstracts and AI-generated summaries, indicating improved readability in the latter (mean Global Readability Score of 36.6±13.8 in the original abstract and of 50.5±11.1 in the AI-generated summary, p<0.001). This trend persisted across abstract topics and journals. While one physician found all summaries correct, the other noted inaccuracies in 32% of cases, with mean rating scores of 4 and 4.7, respectively, and no inter-observer agreement (k value=-0.1). Conclusions ChatGPT demonstrates utility in producing patient-friendly summaries from scientific abstracts in vascular surgery, although the accuracy and quality of AI-generated summaries warrant further scrutiny.
In corso di stampa
0
0
Walter Dorigo, Elena Giacomelli, Cristiano Calvagna*, Filippo Griselli*, Sara Speziali, Aaron Thomas Fargion, Sandro Lepidi**, Raffaele Pulli, Mario D...espandi
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in FLORE sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificatore per citare o creare un link a questa risorsa: https://hdl.handle.net/2158/1388155
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact