Article type
Abstract
Introduction
Plain-language summaries (PLSs) make technical information more accessible and understandable to the general public, overcoming the barrier imposed by technical language to facilitate consumers' and patients' health literacy and active participation in their healthcare
We aim to evaluate the effectiveness of PLSs with large language models (LLM).
Methods
Selection of Systematic Reviews (SRs):
* A sample of 50 SRs published during 2024 regarding diverse topics will be selected from the Cochrane Database of Systematic Reviews.
Development of LLM Prompts:
* A set of standardized prompts will be drafted to instruct LLMs in generating texts following the III.S2 Supplementary material: Guidance for writing Cochrane PLSs.
Testing and Validation of LLM PLSs:
-The developed prompts will be tested on a random subset of 20 SR abstracts to assess and refine their efficacy.
* The remaining 30 SR abstracts will be used for large-scale LLM-generated summaries.
* Internal validation will occur through discussions with the research team, evaluating the summaries for accuracy, understandability, and fidelity to the original published versions of SRs.
* External validation will involve collaboration with patient associations. Fifty randomly selected summaries will be presented to representatives through structured surveys.
Template and Summary Creation:
* Based on the validated prompts and LLM performance, a short template will be developed for crafting patient-friendly summaries.
* LLM-generated summaries will be edited and adapted using the template to ensure consistency and quality.
* The final PLSs will be tailored to the specific characteristics of each SR and its target audience.
Data Analysis:
* Data from the testing, validation processes, and survey will be analyzed quantitatively and qualitatively.
* Quantitative analysis will compare readability scores, understandability ratings, and agreement between different groups of evaluators.
* Qualitative analysis will explore feedback on the PLS's clarity, relevance, and helpfulness compared to the original PLSs.
This comprehensive methodology aims to rigorously test, validate, and refine the approach for generating patient-friendly summaries from systematic reviews using LLMs. The final evaluation with patient associations will provide crucial insights into this method's real-world impact and potential for empowering patients and enhancing health literacy.
Plain-language summaries (PLSs) make technical information more accessible and understandable to the general public, overcoming the barrier imposed by technical language to facilitate consumers' and patients' health literacy and active participation in their healthcare
We aim to evaluate the effectiveness of PLSs with large language models (LLM).
Methods
Selection of Systematic Reviews (SRs):
* A sample of 50 SRs published during 2024 regarding diverse topics will be selected from the Cochrane Database of Systematic Reviews.
Development of LLM Prompts:
* A set of standardized prompts will be drafted to instruct LLMs in generating texts following the III.S2 Supplementary material: Guidance for writing Cochrane PLSs.
Testing and Validation of LLM PLSs:
-The developed prompts will be tested on a random subset of 20 SR abstracts to assess and refine their efficacy.
* The remaining 30 SR abstracts will be used for large-scale LLM-generated summaries.
* Internal validation will occur through discussions with the research team, evaluating the summaries for accuracy, understandability, and fidelity to the original published versions of SRs.
* External validation will involve collaboration with patient associations. Fifty randomly selected summaries will be presented to representatives through structured surveys.
Template and Summary Creation:
* Based on the validated prompts and LLM performance, a short template will be developed for crafting patient-friendly summaries.
* LLM-generated summaries will be edited and adapted using the template to ensure consistency and quality.
* The final PLSs will be tailored to the specific characteristics of each SR and its target audience.
Data Analysis:
* Data from the testing, validation processes, and survey will be analyzed quantitatively and qualitatively.
* Quantitative analysis will compare readability scores, understandability ratings, and agreement between different groups of evaluators.
* Qualitative analysis will explore feedback on the PLS's clarity, relevance, and helpfulness compared to the original PLSs.
This comprehensive methodology aims to rigorously test, validate, and refine the approach for generating patient-friendly summaries from systematic reviews using LLMs. The final evaluation with patient associations will provide crucial insights into this method's real-world impact and potential for empowering patients and enhancing health literacy.