Article type
Abstract
Background:
Though peer review is regarded as the cornerstone of maintaining research integrity, there are some inherent limitations.
Objectives:
To determine to what extent systematic reviewers and post-preprint and post-publication peer review (PPPR) identified methodological and reporting issues of COVID-19 trials that could be easily resolved by authors.
Methods:
Data were derived from the COVID-NMA living systematic review (covid-nma.com). We considered randomized controlled trials (RCTs) evaluating pharmacological treatment for COVID-19 and retrieved risk of bias and outcome reporting bias assessments conducted by systematic reviewers. We also searched for commentary data from PubPeer and preprint servers up to November 6, 2023. We employed thematic analysis to develop themes and domains of methodological and reporting issues identified by commenters.
Results:
We found 500 eligible RCT reports from the database search. Systematic reviewers identified methodological and reporting issues in 446 (89%) RCT reports. In 391 (78%) RCT reports, the issues could be easily resolved by the trial authors, i.e., incomplete reporting (49%), selection of the reported results (52%), and no access to the prespecified plan (25%). Alternatively, 74 (15%) RCT reports received at least one comment on PubPeer or preprint servers, totaling 345 comments. In 46 (9%) RCT reports, the issues identified by post-preprint and PPPR comments could be easily resolved by trial authors, i.e., incomplete reporting (5%), error (4%), statistical analysis (2%), spin (2%), selection of the reported results (1%), and no access to the raw data/prespecified plan (1%).
Conclusions:
Without changing their process, systematic reviewers identified issues in most RCTs that could be easily resolved by trial authors to supplement formal peer review and produce more robust research dissemination. However, the lack of established feedback to authors represents a wasted opportunity for facilitating improvements to manuscript quality overall. Conversely, despite the existing feedback loop to authors present in PPPR, it demonstrated limited effectiveness in identifying methodological and reporting issues of COVID-19 RCTs.
Patient, public, and/or health care consumer involvement: None
Though peer review is regarded as the cornerstone of maintaining research integrity, there are some inherent limitations.
Objectives:
To determine to what extent systematic reviewers and post-preprint and post-publication peer review (PPPR) identified methodological and reporting issues of COVID-19 trials that could be easily resolved by authors.
Methods:
Data were derived from the COVID-NMA living systematic review (covid-nma.com). We considered randomized controlled trials (RCTs) evaluating pharmacological treatment for COVID-19 and retrieved risk of bias and outcome reporting bias assessments conducted by systematic reviewers. We also searched for commentary data from PubPeer and preprint servers up to November 6, 2023. We employed thematic analysis to develop themes and domains of methodological and reporting issues identified by commenters.
Results:
We found 500 eligible RCT reports from the database search. Systematic reviewers identified methodological and reporting issues in 446 (89%) RCT reports. In 391 (78%) RCT reports, the issues could be easily resolved by the trial authors, i.e., incomplete reporting (49%), selection of the reported results (52%), and no access to the prespecified plan (25%). Alternatively, 74 (15%) RCT reports received at least one comment on PubPeer or preprint servers, totaling 345 comments. In 46 (9%) RCT reports, the issues identified by post-preprint and PPPR comments could be easily resolved by trial authors, i.e., incomplete reporting (5%), error (4%), statistical analysis (2%), spin (2%), selection of the reported results (1%), and no access to the raw data/prespecified plan (1%).
Conclusions:
Without changing their process, systematic reviewers identified issues in most RCTs that could be easily resolved by trial authors to supplement formal peer review and produce more robust research dissemination. However, the lack of established feedback to authors represents a wasted opportunity for facilitating improvements to manuscript quality overall. Conversely, despite the existing feedback loop to authors present in PPPR, it demonstrated limited effectiveness in identifying methodological and reporting issues of COVID-19 RCTs.
Patient, public, and/or health care consumer involvement: None