Article type
Abstract
Background: Systematic reviews (SRs) are designed to inform evidence-based practice. Risk of Bias (ROB) tables, an integral component of SRs, are crucial for determining biases. However, non-adherence to robust research designs diminishes the certainty of evidence produced by SRs. Studying ROB tables reveals trends and highlights problematic design practices.
Objectives: To determine which design elements are not consistently present in ROB tables within effectiveness SRs.
Methods: We conducted a cross-sectional study on ROB tables in effectiveness SRs, searching JBI Database of Systematic Reviews and Implementation Reports/JBI Evidence Synthesis from 2011 to present. Included SRs used the older, pseudo-randomized critical appraisal tool, or the 2020 randomized control trial (RCT) tool. Using SPSS 29, we validated the published percent of criteria met for each question, calculated the total percent of criteria met per study, and computed the average quality of all included studies in each ROB table.
Results: (N = 96) ROB tables representing 771 individual studies. RCT risk of bias tables (n = 30) had moderate study quality (69%), due to four study design questions about blinding (Q 2, 4–6), with ≤ 45% “yes” response. Pseudo-randomized ROB tables (n = 66) demonstrated lower overall study quality (66%), with four problematic questions (Q 2–5) having ≤ 45% “yes” response, including three on blinding and one on intention-to-treat analysis. Low scoring questions indicate potential biases related to; a) study administration, b) selection and allocation, c) participant retention, and d) assessment, detection and measurement of outcomes, threatening internal validity.
Additionally, we found inconsistent practices in the scoring system of ROB tables calculated by reviewers. Some SR authors assigned more points to “yes” and less to all other answers, changed the denominator when a critical appraisal question was “unclear/no/NA”, or in rare instances, added excluded studies in the calculations. These inconsistencies inflated percentages.
Conclusion: Results demonstrate a heightened need for researchers to make every attempt to create strong studies by adhering to rigorous design elements. Threats to internal validity impact conclusions regardless of statistical analyses conducted. Thorough explanation of design elements excluded may enhance reviewers, editors, and research consumers’ understanding of true study quality.
Objectives: To determine which design elements are not consistently present in ROB tables within effectiveness SRs.
Methods: We conducted a cross-sectional study on ROB tables in effectiveness SRs, searching JBI Database of Systematic Reviews and Implementation Reports/JBI Evidence Synthesis from 2011 to present. Included SRs used the older, pseudo-randomized critical appraisal tool, or the 2020 randomized control trial (RCT) tool. Using SPSS 29, we validated the published percent of criteria met for each question, calculated the total percent of criteria met per study, and computed the average quality of all included studies in each ROB table.
Results: (N = 96) ROB tables representing 771 individual studies. RCT risk of bias tables (n = 30) had moderate study quality (69%), due to four study design questions about blinding (Q 2, 4–6), with ≤ 45% “yes” response. Pseudo-randomized ROB tables (n = 66) demonstrated lower overall study quality (66%), with four problematic questions (Q 2–5) having ≤ 45% “yes” response, including three on blinding and one on intention-to-treat analysis. Low scoring questions indicate potential biases related to; a) study administration, b) selection and allocation, c) participant retention, and d) assessment, detection and measurement of outcomes, threatening internal validity.
Additionally, we found inconsistent practices in the scoring system of ROB tables calculated by reviewers. Some SR authors assigned more points to “yes” and less to all other answers, changed the denominator when a critical appraisal question was “unclear/no/NA”, or in rare instances, added excluded studies in the calculations. These inconsistencies inflated percentages.
Conclusion: Results demonstrate a heightened need for researchers to make every attempt to create strong studies by adhering to rigorous design elements. Threats to internal validity impact conclusions regardless of statistical analyses conducted. Thorough explanation of design elements excluded may enhance reviewers, editors, and research consumers’ understanding of true study quality.