Article type
Abstract
Background:
Mistrust in research is increasing, causing some to argue that relying solely on publications or summary-level trial data is no longer sufficient to inform health care decision-making. The availability of individual participant data (IPD) enables more comprehensive integrity checks, but guidance on which checks to use and how to conduct them is lacking.
Objective:
We aimed to address this by developing a tool for assessing the integrity of randomized controlled trials (RCTs) using IPD.
Methods:
We conducted a literature review to identify existing methods to assess the integrity of RCTs and their IPD. These were mapped into key domains and discussed among an expert advisory group. Agreed items were operationalized in a standardized tool and automated when possible. We piloted this tool in 73 trials from 2 IPD meta-analyses and conducted preliminary validation in a sample of 5 trials with IPD datasets flagged by journal editors as having known integrity issues and 8 similar datasets without known integrity issues. Evaluation workshops informed iterative refinements of the tool.
Results:
We developed the IPD Integrity Tool (Figure 1), comprising 31 items across 7 aggregate data-level and 8 IPD-specific domains. IPD-specific domains include unusual data patterns, issues with baseline characteristics, lack of expected correlations, date violations, unusual allocation patterns, internal/external inconsistencies, and plausibility of data. Users rate each item as having either no issues, some/minor issue(s), or many/major issue(s) according to a decision guide, and justification for each rating is recorded. If there are many and/or major issues that cannot be resolved, the study should be excluded from evidence synthesis and/or not considered suitable for publication. In our validation checks, the tool accurately identified all 5 studies with known integrity issues.
Conclusions:
The IPD Integrity Tool enables users to assess the integrity of RCTs via examination of IPD. The Tool may be applied by various stakeholders, including journal editors to prevent publication of untrustworthy studies and systematic reviewers to assess RCTs for inclusion in analyses.
Relevance and importance to patients:
Implementation of the IPD Integrity Tool can improve trustworthiness of the evidence base that informs patient care, leading to improved health outcomes.
Mistrust in research is increasing, causing some to argue that relying solely on publications or summary-level trial data is no longer sufficient to inform health care decision-making. The availability of individual participant data (IPD) enables more comprehensive integrity checks, but guidance on which checks to use and how to conduct them is lacking.
Objective:
We aimed to address this by developing a tool for assessing the integrity of randomized controlled trials (RCTs) using IPD.
Methods:
We conducted a literature review to identify existing methods to assess the integrity of RCTs and their IPD. These were mapped into key domains and discussed among an expert advisory group. Agreed items were operationalized in a standardized tool and automated when possible. We piloted this tool in 73 trials from 2 IPD meta-analyses and conducted preliminary validation in a sample of 5 trials with IPD datasets flagged by journal editors as having known integrity issues and 8 similar datasets without known integrity issues. Evaluation workshops informed iterative refinements of the tool.
Results:
We developed the IPD Integrity Tool (Figure 1), comprising 31 items across 7 aggregate data-level and 8 IPD-specific domains. IPD-specific domains include unusual data patterns, issues with baseline characteristics, lack of expected correlations, date violations, unusual allocation patterns, internal/external inconsistencies, and plausibility of data. Users rate each item as having either no issues, some/minor issue(s), or many/major issue(s) according to a decision guide, and justification for each rating is recorded. If there are many and/or major issues that cannot be resolved, the study should be excluded from evidence synthesis and/or not considered suitable for publication. In our validation checks, the tool accurately identified all 5 studies with known integrity issues.
Conclusions:
The IPD Integrity Tool enables users to assess the integrity of RCTs via examination of IPD. The Tool may be applied by various stakeholders, including journal editors to prevent publication of untrustworthy studies and systematic reviewers to assess RCTs for inclusion in analyses.
Relevance and importance to patients:
Implementation of the IPD Integrity Tool can improve trustworthiness of the evidence base that informs patient care, leading to improved health outcomes.