Background Formulating recommendations in developing practice guidelines involves complex and rigorous processes. Therefore, more resources and expertise are needed. Artificial intelligence (AI) is promising in accelerating the guideline development process. This study aims to assess the feasibility of three large language models (ChatGPT-3.5, Claude, and Bard) in generating guideline recommendations, evaluate their concordance among the generated recommendations, and further explore the feasibility of AI-assisted evidence-based decision-making.
Methods The general and specific prompts of the three large language models were drafted and validated. We searched Embase, Web of Science, PubMed, and guidelines websites to include evidence-based guidelines related to health and lifestyle. We randomly selected recommendations from the included guidelines as the sample and extracted the evidence base supporting the selected recommendations. The prompts and evidence were fed into three AI systems to generate structured recommendations.
Results ChatGPT-3.5 exhibited the highest proficiency in comprehensively extracting and amalgamating evidence information to formulate novel insights. Bard consistently adhered to guidelines, aligning its algorithm closely with existing guidelines' intrinsic principles. Conversely, Claude tended to generate fewer topics, focusing on evidence analysis and mitigating the risk of extraneous and irrelevant information. Among the six recommendations generated in this study, the average consistency ranges from 50% to 100%.
Conclusions The findings suggest that AI can expedite the formulation of recommendations in developing practice guidelines. While AI can enhance efficiency, its role should be complementary rather than substitutive, working in tandem with medical professionals in guideline development.