Background: : ChatGPT is a language model developed by OpenAI that is trained to generate human-like text based on large amounts of data and has the potential for role-playing during informed decisions. Objective: To compare the accuracy of an artificial intelligence chatbot to clinical practice guidelines (CPGs) recommendations for providing answers to complex clinical questions on lumbosacral radicular pain.
Methods: We extracted recommendations from recent CPGs for diagnosing and treating lumbosacral radicular pain. Relative clinical questions were developed and queried to Open AI’s ChatGPT (GPT-3.5). We compared ChatGPT answers to CPGs recommendations by assessing the (i) internal consistency of ChatGPT answers by measuring the percentage of text wording similarity when a clinical question was posed three times, (ii) reliability between two independent reviewers in grading ChatGPT answers, and (iii) accuracy of ChatGPT answers compared to CPGs recommendations. Reliability was estimated using Fleiss' kappa (κ) coefficients, and accuracy by inter-observer agreement as the frequency of the agreements among all judgements.
Results: We tested nine clinical questions (table 1). The internal consistency of text ChatGPT answers was unacceptable across all three trials in all clinical questions (mean percentage of 49%, standard deviation of 15). Intra (reviewer 1: κ=0.90 standard error (SE) =0.09; reviewer 2: κ=0.90 se=0.10) and inter-reliability (κ=0.85 SE=0.15) between the two reviewers was “almost perfect”. Accuracy between ChatGPT answers and CPGs recommendations was slight, demonstrating agreement in 33% of recommendations (table 2).
Conclusions: ChatGPT performed poorly in internal consistency and accuracy of the indications generated compared to clinical practice guideline recommendations for lumbosacral radicular pain.