Article type
Abstract
"Background:
Financial conflict of interest (FCOI) in research is common and may impact trial design and results. The Tool for Addressing Conflict of Interest in Trials (TACIT) enables evidence synthesis researchers to systematically evaluate FCOI of included studies but adds effort to the review workflow. The objective of this proof-of-concept study is to assess the performance of a large language model (LLM) in assessing FCOI in published studies using TACIT.
Methods:
We developed the study protocol using a preliminary version of TACIT presented at the 2023 Cochrane Colloquium and identified 75 data elements (20 categorical, 55 free text) corresponding to signaling questions. We used a convenience sample of 13 published studies included in previously conducted systematic reviews: 7 randomized controlled trials (RCTs) of drugs, 2 RCTs of implementation interventions, and 4 non-randomized studies of interventions. We used the browser version of the Claude 2.0 LLM to upload the portable document format of each study article and associated materials and prompt the LLM for each TACIT data element. One researcher reviewed the LLM output for accuracy and completeness (LLM-assisted process). In parallel, two independent researchers conducted FCOI assessments with TACIT and adjudicated responses with each other (standard process). We tracked the time required for both and compared information from the LLM-assisted and standard process to determine concordance, classify errors, and look for patterns based on study design and funding source.
Results:
Our analysis is ongoing and will be completed by June 2024. Five studies were funded by industry (e.g., pharmaceutical company), 6 were funded by non-industry (e.g., government agency), 1 reported no funding, and 1 did not report funding.
Discussion:
This study will provide an early indication of whether Claude 2.0 can be used to semi-automate FCOI judgments of published research studies. If there is considerable agreement between the LLM-assisted and manual FCOI assessments, a study within reviews is the next step to evaluate an LLM-assisted process within the real-time workflow of a systematic review. Integrating FCOI judgments into review workflows using LLMs could increase efficiency for reviewers and provide more transparency about the evidence to patients and clinicians."
Financial conflict of interest (FCOI) in research is common and may impact trial design and results. The Tool for Addressing Conflict of Interest in Trials (TACIT) enables evidence synthesis researchers to systematically evaluate FCOI of included studies but adds effort to the review workflow. The objective of this proof-of-concept study is to assess the performance of a large language model (LLM) in assessing FCOI in published studies using TACIT.
Methods:
We developed the study protocol using a preliminary version of TACIT presented at the 2023 Cochrane Colloquium and identified 75 data elements (20 categorical, 55 free text) corresponding to signaling questions. We used a convenience sample of 13 published studies included in previously conducted systematic reviews: 7 randomized controlled trials (RCTs) of drugs, 2 RCTs of implementation interventions, and 4 non-randomized studies of interventions. We used the browser version of the Claude 2.0 LLM to upload the portable document format of each study article and associated materials and prompt the LLM for each TACIT data element. One researcher reviewed the LLM output for accuracy and completeness (LLM-assisted process). In parallel, two independent researchers conducted FCOI assessments with TACIT and adjudicated responses with each other (standard process). We tracked the time required for both and compared information from the LLM-assisted and standard process to determine concordance, classify errors, and look for patterns based on study design and funding source.
Results:
Our analysis is ongoing and will be completed by June 2024. Five studies were funded by industry (e.g., pharmaceutical company), 6 were funded by non-industry (e.g., government agency), 1 reported no funding, and 1 did not report funding.
Discussion:
This study will provide an early indication of whether Claude 2.0 can be used to semi-automate FCOI judgments of published research studies. If there is considerable agreement between the LLM-assisted and manual FCOI assessments, a study within reviews is the next step to evaluate an LLM-assisted process within the real-time workflow of a systematic review. Integrating FCOI judgments into review workflows using LLMs could increase efficiency for reviewers and provide more transparency about the evidence to patients and clinicians."