Performance of ChatGPT compared to clinical practice guidelines in making informed decisions for Lumbosacral Radicular Pain

Poster
2024 Prague [Global Evidence Summit]
Gianola S1, Bargeri S1, Castellini G1, Cook C2, Palese A3, Pillastrini P4, Salvalaggio S5, Turolla A4, Rossettini G6
1 Unit of Clinical Epidemiology, IRCCS Ospedale Galeazzi Sant'Ambrogio, Milano, Italy
2Department of Orthopaedics, Duke University, Durham, NC, USA
3Department of Medical Sciences, University of Udine, Udine, Italy
4Department of Biomedical and Neuromotor Sciences (DIBINEM), Alma Mater University of Bologna, Bologna, Italy; Unit of Occupational Medicine, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
5Laboratory of Computational Neuroimaging, IRCCS San Camillo Hospital, Venice, Italy; Padova Neuroscience Center, Università degli Studi di Padova, Padova, Italy
6School of Physiotherapy, University of Verona, Verona, Italy

Background:

: ChatGPT is a language model developed by OpenAI that is trained to generate human-like text based on large amounts of data and has the potential for role-playing during informed decisions.

Objective:

To compare the accuracy of an artificial intelligence chatbot to clinical practice guidelines (CPGs) recommendations for providing answers to complex clinical questions on lumbosacral radicular pain.

Methods:

We extracted recommendations from recent CPGs for diagnosing and treating lumbosacral radicular pain. Relative clinical questions were developed and queried to Open AI’s ChatGPT (GPT-3.5). We compared ChatGPT answers to CPGs recommendations by assessing the (i) internal consistency of ChatGPT answers by measuring the percentage of text wording similarity when a clinical question was posed three times, (ii) reliability between two independent reviewers in grading ChatGPT answers, and (iii) accuracy of ChatGPT answers compared to CPGs recommendations. Reliability was estimated using Fleiss' kappa (κ) coefficients, and accuracy by inter-observer agreement as the frequency of the agreements among all judgements.

Results:

We tested nine clinical questions (table 1). The internal consistency of text ChatGPT answers was unacceptable across all three trials in all clinical questions (mean percentage of 49%, standard deviation of 15). Intra (reviewer 1: κ=0.90 standard error (SE) =0.09; reviewer 2: κ=0.90 se=0.10) and inter-reliability (κ=0.85 SE=0.15) between the two reviewers was “almost perfect”. Accuracy between ChatGPT answers and CPGs recommendations was slight, demonstrating agreement in 33% of recommendations (table 2).

Conclusions:

ChatGPT performed poorly in internal consistency and accuracy of the indications generated compared to clinical practice guideline recommendations for lumbosacral radicular pain.